Change fwload_only to fwload_force.

Looks odd to me, but OK.

In D45660#1042762, @mav wrote:

In D45660#1042742, @ken wrote:

So here is what the debugging log message in isp_getpdb() shows. isp0 and isp1 are connected to LTO-6 tape drives via an 8Gb switch. isp2 is directly connected to an LTO-6 in loop mode:

isp0: Chan 0 handle 0x0 Port 0xfffc01 flags 0x0 curstate 77 laststate 77
isp0: Chan 0 handle 0x1 Port 0x011b26 flags 0x40a0 curstate 46 laststate 46
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp1: Chan 0 handle 0x0 Port 0xfffc01 flags 0x0 curstate 77 laststate 77
isp1: Chan 0 handle 0x1 Port 0x011a26 flags 0x40a0 curstate 46 laststate 46
isp1: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp1: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp2: Chan 0 handle 0x0 Port 0x000026 flags 0x40a0 curstate 46 laststate 46

So it seems the upper 4 bits can actually have different values, not some "non-NVMe" constant, as I thought. I am OK with this patch if it solves the issue, but it would definitely be good to understand what it is, in case there is something important.

In D45660#1042742, @ken wrote:

So here is what the debugging log message in isp_getpdb() shows. isp0 and isp1 are connected to LTO-6 tape drives via an 8Gb switch. isp2 is directly connected to an LTO-6 in loop mode:

isp0: Chan 0 handle 0x0 Port 0xfffc01 flags 0x0 curstate 77 laststate 77
isp0: Chan 0 handle 0x1 Port 0x011b26 flags 0x40a0 curstate 46 laststate 46
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp1: Chan 0 handle 0x0 Port 0xfffc01 flags 0x0 curstate 77 laststate 77
isp1: Chan 0 handle 0x1 Port 0x011a26 flags 0x40a0 curstate 46 laststate 46
isp1: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp1: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44
isp2: Chan 0 handle 0x0 Port 0x000026 flags 0x40a0 curstate 46 laststate 46

In D45688#1042676, @mav wrote:

It seems a good tunable, except I am not getting the meaning of "only" there. Why not "always", "force" or something like that?

In D45660#1042675, @mav wrote:

None of QLogic documents I have know nothing about NVMe, and this state field is declared is byte there. I have no objections for this patch, but a bit curios what NVMe status do we see there for non-NVMe devices.

It seems a good tunable, except I am not getting the meaning of "only" there. Why not "always", "force" or something like that?

None of QLogic documents I have know nothing about NVMe, and this state field is declared is byte there. I have no objections for this patch, but a bit curios what NVMe status do we see there for non-NVMe devices.

In D35784#812281, @mav wrote:

Without this it does not explicitly hold boot, but it blocks GEOM event thread on taste attempt, which also blocks boot, but also blocks most of GEOM commands.

Without this it does not explicitly hold boot, but it blocks GEOM event thread on taste attempt, which also blocks boot, but also blocks most of GEOM commands.

At this point, I'm only worried about da's state machine holding up boot now where it didn't used to do so... And I'm not sure, exactly, how to measure that worry ...

Remove incorrect comment.

Move softc->state = NDA_STATE_NORMAL; before disk_create().

Update after Warner's comments.

I generally like this, even though I had a bunch of nit-picky comments and questions.
I only see one real issue for sure, but maybe a second race (but maybe not, I'm asking to make sure it can't happen).

Remove unneeded now disk_attr_changed() call.

Restore commas, fix compat shims.

If there is some use high priority, then it works for SATA and it is simple, but since it is absolute priority, the difference between normal and high is too big to use it without very good reason. For low priority though, which would be useful for background operations even with absolute priorities, I haven't found a working implementation so far, unless potentially NVMe. I am hoping to get some comments from ${HDD vendor} about it.

Have we reached any conclusions about whether to do any of the ideas suggested in this phabricator thread?

Just for information, I've also experimented with isochronous NCQ priority (AKA NCQ streaming). I hoped that setting large timeout would reduce the request priority. But at least on WD Red I see no any priority effects until the timeout is reached, and I see priority increase (again with the IOPS problem) when it is. It is good to see that the feature is really working, but unfortunately I see no usage for it in this shape. I see plenty of use cases for low priority (that SATA/SAS drives don't provide), but not really for high priority (that they do, but not very efficiently). NVMe seems to have usable priority concept and some devices support it, just not sure how important is the priority for pretty fast NVMe's.

I've tried opposite approach of adding LOWPRIO flag instead and using it only for background operations in few places, and marking BIOs without it high-priority in ATA/SCSI. But while testing it I've noticed that disk random IOPS drop to almost non-NCQ level on a mix of different priorities. And I am measuring the same on both WD and HGST. I don't understand what is going on there, may be I am missing something, but that is unacceptable trade-off to me. I've uploaded my present patch in case somebody wish to play, but probably won't commit it in this state.

Priority is working on top of tag, affecting specifically only commands tagged as SIMPLE . The ORDERED and HEAD tags still have their function as they are mandatory in their fencing semantics, while priority is a softer hint for a schduler.