Looks good. Few places that call it also check for G_PF_WITHER, but they are using g_post_event() for it, that opens a race window.

I am happy that after all the years the transition is finally done. :)

In D27225#608119, @kib wrote:

I would worry about reserved KVA misalignment then. Right now, I believe, pbuf' allocated KVA is naturally aligned.

It looks great! Thanks!

In D27225#607878, @imp wrote:

Does ZFS need any tweaking? I know it's mildly wrong today in scheduling 1MB I/Os today, but at least GEOM will break that up so the I/Os from there on down will be legal.

If there is some use high priority, then it works for SATA and it is simple, but since it is absolute priority, the difference between normal and high is too big to use it without very good reason. For low priority though, which would be useful for background operations even with absolute priorities, I haven't found a working implementation so far, unless potentially NVMe. I am hoping to get some comments from ${HDD vendor} about it.

Just for information, I've also experimented with isochronous NCQ priority (AKA NCQ streaming). I hoped that setting large timeout would reduce the request priority. But at least on WD Red I see no any priority effects until the timeout is reached, and I see priority increase (again with the IOPS problem) when it is. It is good to see that the feature is really working, but unfortunately I see no usage for it in this shape. I see plenty of use cases for low priority (that SATA/SAS drives don't provide), but not really for high priority (that they do, but not very efficiently). NVMe seems to have usable priority concept and some devices support it, just not sure how important is the priority for pretty fast NVMe's.

I've tried opposite approach of adding LOWPRIO flag instead and using it only for background operations in few places, and marking BIOs without it high-priority in ATA/SCSI. But while testing it I've noticed that disk random IOPS drop to almost non-NCQ level on a mix of different priorities. And I am measuring the same on both WD and HGST. I don't understand what is going on there, may be I am missing something, but that is unacceptable trade-off to me. I've uploaded my present patch in case somebody wish to play, but probably won't commit it in this state.

Priority is working on top of tag, affecting specifically only commands tagged as SIMPLE . The ORDERED and HEAD tags still have their function as they are mandatory in their fencing semantics, while priority is a softer hint for a schduler.

More experiments with SATA WD REDs show that priorities there more like absolute with deadline. On WD20EFRX-68E on heavy random workload I see low-priority requests in presence of high-priority are all delayed for about a second, while on WD80EFZX-68U they are all delayed for about 5 seconds. So big difference makes me think it is unusable for differentiation of sync vs async requests, but should still be good for read/write vs scrub/initialization/etc differentiation. Unfortunately I still haven't found any capable SAS drive to check there, but considering SATL directly map one into another I suppose they should have the same (absolute) semantics.

Looks good to me.

Thanks!

I have no objections, but the link specified lead me to Linux' drivers/pci/quirks.c file, where I've found the same done for 12 different Marvell HBAs. Why not do the same?

The core around this looks pretty weird to me, considering hpt_async() immediately calls panic(). But I don't expect troubles from this change.