bhyve/virtio-scsi: Preallocate all I/O requests

[rosenfeld_grumpf.hope-2000.org]

Summary

By preallocating all I/O requests on all queues, we can take most
allocations out of the hot I/O code paths and simplify the code
significantly. While here, make sure we check all allocations for
success and make sure to handle failures gracefully.

Additionally, check for I/O request validity as early as possible,
and return illegal requests immediately.

[rosenfeld_grumpf.hope-2000.org]

Rebase.

[rosenfeld_grumpf.hope-2000.org]

pci_vtscsi_proc()

[rosenfeld_grumpf.hope-2000.org]

Rebase.

[rosenfeld_grumpf.hope-2000.org]

Sync with latest illumos bits after code review.

Use a size_t type where appropriate instead of the abomination that is uint.

[markj]

This broadly looks ok, I have a few inline comments.

usr.sbin/bhyve/pci_virtio_scsi.c

141

Some comment explaining how these three mutexes are used would be appreciated.

603

Is the vsr_queue field used for anything?

705

Is it really necessary to clear the whole request structure like this?

874

There's no cleanup of the threads in this error path. If you allocate all of the worker structures at once, before the loop, then you don't need to think about cleaning up in this error path (aside from the requests allocated in the loop above, which are easy to deal with).

[rosenfeld_grumpf.hope-2000.org]

Address review from markj.

[rosenfeld_grumpf.hope-2000.org]

usr.sbin/bhyve/pci_virtio_scsi.c

603

Looks like it isn't used anywhere by the code, but I found it useful during debugging, especially with multiple I/O queue support.

705

I don't know whether it's strictly necessary, but I feel much better knowing there's no stale data left in there before we reuse it, especially not in any of the iovecs.

[rosenfeld_grumpf.hope-2000.org]

Rebase, and add cleanup code to go with all the new checks for success.