bhyve/virtio-scsi: Preallocate all I/O requests

[rosenfeld_grumpf.hope-2000.org]

Summary

By preallocating all I/O requests on all queues, we can take most
allocations out of the hot I/O code paths and simplify the code
significantly. While here, make sure we check all allocations for
success and make sure to handle failures gracefully.

Additionally, check for I/O request validity as early as possible,
and return illegal requests immediately.

[rosenfeld_grumpf.hope-2000.org]

Rebase.

[rosenfeld_grumpf.hope-2000.org]

pci_vtscsi_proc()

[rosenfeld_grumpf.hope-2000.org]

Rebase.

[rosenfeld_grumpf.hope-2000.org]

Sync with latest illumos bits after code review.

Use a size_t type where appropriate instead of the abomination that is uint.

[markj]

This broadly looks ok, I have a few inline comments.

usr.sbin/bhyve/pci_virtio_scsi.c

141

Some comment explaining how these three mutexes are used would be appreciated.

602

Is the vsr_queue field used for anything?

699

Is it really necessary to clear the whole request structure like this?

873

There's no cleanup of the threads in this error path. If you allocate all of the worker structures at once, before the loop, then you don't need to think about cleaning up in this error path (aside from the requests allocated in the loop above, which are easy to deal with).

[rosenfeld_grumpf.hope-2000.org]

Address review from markj.

[rosenfeld_grumpf.hope-2000.org]

usr.sbin/bhyve/pci_virtio_scsi.c

602

Looks like it isn't used anywhere by the code, but I found it useful during debugging, especially with multiple I/O queue support.

699

I don't know whether it's strictly necessary, but I feel much better knowing there's no stale data left in there before we reuse it, especially not in any of the iovecs.