Fixing things post-commit is much better than forcing the work to languish. This review has been plagued by poor participation for far too long, and it's reflecting poorly on the FreeBSD community. I'm ok with giving people until Wednesday, but after that it goes in, love it or hate it.

There has been no dissension on this review since early November, and HPS has done a lot to update the code to address the comments from then. Given the lack of new input, I recommend that this code be committed.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

Add safety belts for page alignment and page crossing.

Address feedback from Jim

Remove a spurious printf

Last comment might have been lost. I've set the default to 1.

Address string typo and move a structure field around to a more logical spot.

I'll set the default to 1 in the next revision.

Since the goal here is to take 'struct device' out of the public FreeBSD kernel API, it should be renamed to "_device" instead of "device_"

Actually I would set the default user value to 0xffffffff, and then do a min() calculation with that and the calculated value. That way you never go above the calculated value, but you can go lower (maybe clip the lower value to something sensible, not allow absurdly low values like '1').

Sounds good

Smaller than the calculated value is what I meant. It's a convenient backstop against unexpected code, disk, and controller bugs. Just a simple cpi->maxio = min(sc->user_max_io, cpi->maxio);

Looks good in principle. Would it be possible to have this value be overridable by a user tunable/sysctl?

Agreed on the MFC proposal.

Thanks!

Remove a line that crept in.

Let's break this down into manageable tasks:

1. Eliminate the use of vtophys, pci_alloc_consistent/pci_map_single, and contigmalloc/contigfree. Everything should be done with busdma. For the sgl and sense buffers, they should be allocated in a block via bus_dmamem_alloc() and then split out to the ctx objects, with their physaddr pre-computed.
2. Switch linked lists from linux macros to *BSD queue.h macros.
3. Get rid of the use of the rest of the linux shims, like create_workqueue/destroy_workqueue.
4. Get rid of the linux typedefs. Is there an intention to share vow_pvscsi.h across OS's? If so then let's see if we can keep the types isolated to that.
5. Everything else that Ngie said.

A couple of key things:

1. Licenses must be clarified. This can't go into the freebsd tree without that.
2. It looks like this is a mix of linux and freebsd. I get that the original code was linux, and effort was made to shim in convenient places. However, it's a mixed up mess now. Either keep the linux sources as unaltered as possible and put all of the shimming into separate files, or let's work on making this use native APIs. Keeping it in-between makes it hard to maintain and hard to adopt future changes from VMWare. Do you expect this code to stay alive and evolve? The last thing I want is for the code to be tossed in and then abandoned, leaving it hard to maintain and ultimately prone to rot.
3. Can the ISILON blocks be turned into something more descriptive? Do they encapsulate functionality that is completely unique to Isilon that could never be used by others? If so, then why put such code into FreeBSD? Again, it's hard for someone else to review and maintain this code when there are blocks of mystery functionality.

That's not true. device_set_desc and friends are specifically designed to be operated on in device_probe.

I need the changes to mbuf.h

I've made the changes to if_an and committed that part of the patch to freebsd-head

Are the changes to if_an, if_bxe, and kern_mbuf.c intentional?

Fix a compile error from the previous revision.

A few naming tweaks, and add a CAM_SMP_STATUS_ERROR to the list of handled
errors.

In D5943#126878, @asomers wrote:

I like the spirit of this change. My main worry is that a poorly behaved device could generate so many messages that it would slow the system and DOS devd clients. But eliminating that risk might require a whole new replacement for devctl(4).

In D5943#126839, @ken wrote:

See the comment on using the zone allocator.

The question remains, what operations are we looking to track, and why? And what operations are we not looking to track, and why? How do we accomplish filtering?

Switch to use static allocations for sbuf to avoid locking issues. Also
refactor the handling of each error case, and add a few cases that had been
lost along the way.

Lost a small piece of printf formatting in the last revision, restore it.

Removed some extraneous debugging. Made scsi_cdb_string() use on-stack
storage for the sbuf in order to improve reliability.

I agree with the worry over the use of mutexes and allocation failures. I thing I considered was having scsi_cdb_string() take an option for whether to allocate the sbuf off the stack or dynamically. What do you think?

Note that I intend to switch a few consumers of scsi_cdb_string(), notably scsi_command_string() to use this variant in order to streamline the use of sbufs and eliminate the need for stack storage.

I like the general idea a lot. I'd like to improve it further, though. You're basically hiding the tag away from the new API, which is good IMHO, but you don't quite go far enough. For your example:

If the PIM_ATA_EXT flag is getting set by the SIM and checked by the periph, it would be a programming error for the periph to set the ATA_FLAG_AUX flag if PIM_ATA_EXT was not set. Therefore, maybe make the check for ATA_FLAG_AUX be a KASSERT instead of a mandatory runtime check?

My thought was to retire the existing XPT_ATA_IO value and move to XPT_ATA_IO_EXT. I don't disagree with your approach, need to think on it.