if_ntb: Diff-reduce with Linux; add queue index type

Add ntb_q_idx_t so it is more clear which struct members are of the same
type (some bogus uint64_ts snuck in that should have been unsigned int).

Add tx_err_no_buf and s/ENOMEM/EBUSY/ in tx_enqueue to match Linux.

Sponsored by: EMC / Isilon Storage Division