In D8224#170848, @kib wrote:

Do you have specific questions to me, e.g. about the interfaces used ? I definitely would not read the hv daemon code, but sure can answer questions about the supposed use of the FreeBSD interfaces to VFS/VM.

Remove the taskqueue_drain on if_link_status, if_link_state_change should be fixed instead.

In D8071#167026, @hselasky wrote:

Looks good. Do you have access to commit it?

I will commit it today, unless hps beats me to it :)

Disallow destination cpu change for Hyper-V's I/O APIC, suggested by: jhb

In D7949#165633, @sepherosa_gmail.com wrote:

In D7949#165409, @sepherosa_gmail.com wrote:

In D7949#165286, @jhb wrote:

In D7949#165151, @sepherosa_gmail.com wrote:

In D7949#165052, @jhb wrote:

A couple of comments:

1. The shuffle is going to go away (I plan to enable EARLY_AP_STARTUP by default in 12 and possibly 11.x).

2. Is the underlying issue that you don't handle MSI-X interrupt migration correctly? If so, we have a workaround for that case for Xen that you could use which prevents all migration (not just the boot-time migration).

• On Hyper-V MSI-Xs are always allocated after AP started with the WIP Hyper-V PCI-bridge code for PCI passthrough/SR-IOV, so re-shuffle is actually not involved.
• On Hyper-V, only atkbd and ata allocate interrupts from PIC (let's ignore the fdc there :), which are not performance critical at all.

The main issue, we found is that when re-shuffle happened, CAM was issuing commands to ata controller, and the ata interrupt was lost due to the reshuffle, which leaded to all kinds of weirdness later on (mainly because the disk itself is shared w/ the ata controller and the Hyper-V synthetic controller).

So the odd thing there is that in theory the shuffle shouldn't happen during CAM device discovery. The normal device discovery happens during the interrupt hooks run at SI_SUB_INT_CONFIG_HOOKS and thread0 waits for any interrupt hooks to finish before proceeding. The CAM bus scans only clear their hook once the bus scan is finished. However, what may be happening is that GEOM decides to asynchronously kick off tasting for the various partition providers, etc. after the CAM bus scans have finished. Even then, interrupts still shouldn't be lost as interrupt migration for an I/O APIC pin is careful to first mask the pin, then update it with the new (CPU, IDT) pair. After that is done, the old (CPU, IDT) pair is released via apic_free_vector() which uses sched_bind() to move to the old CPU and leave interrupts disabled for at least one instruction to catch any "in-flight" interrupts on the old CPU.

Certain commands are sent after disks are attached asynchronously after the config_hook, AFAIR, writecache and readahead etc.

It might be worth adding a 'DELAY(100)' or so after the sched_bind() to give any in-flight interrupts more time to hit the destination CPU (though the sched_bind() is going to go through an IPI send/ack if we weren't already running on the old CPU).

I am not going to mess up CAM too much this time, since as you have noted EARLY_AP will be enabled by default on 11 and 12. So I believe disabling the reshuffle on Hyper-V is the best practice (well, I also need to MFC it to 10).

So my concern is that if the I/O APIC emulation in Hyper-V doesn't handle shuffling, then that means that users using 'cpuset -x' to move interrupts around from the command line can also lose interrupts. In that case, I think we should disable interrupt migration on Hyper-V in general (not just in the shuffle). However, it would be helpful to know what is actually broken. Xen had a bug with MSI-X in particular such that older versions of Xen didn't handle writes to MSI-X table entries while MSI-X was enabled (but the entry was masked). That affected 'cpuset -x' as well as the boot round-robin, so for Xen we disable all MSI-X migration on hypervisors with the bug.

I will check Dexuan about MSI-X migration, since PCI passthrough is kinda working (we get a working passthrough-ed ix w/ MSI-X). However, if I recall correctly, he told me MSI-X migration works w/ 'cpuset -x' (not under heavy load through). So currently I believe we only need to prevent line based interrupts migration, which really goes through IOAPIC.

Confirmed w/ Dexuan, cpuset -x work for PCI passthrough'ed MSI-X. So we only need to disable line interrupt migration.

In D7693#163011, @sepherosa_gmail.com wrote:

In D7693#160050, @mav wrote:

One more downside of this CAM modification approach is that if previously user not loading hyperv drivers automatically got legacy emulation ATA, after this change -- he will get no disks at all.

How about the latest patch? The changes to the CAM (5 lines of effective code) is much less invasive than previous ones, and I think it address various concerns properly.

In D7949#165409, @sepherosa_gmail.com wrote:

In D7949#165286, @jhb wrote:

In D7949#165151, @sepherosa_gmail.com wrote:

In D7949#165052, @jhb wrote:

A couple of comments:

1. The shuffle is going to go away (I plan to enable EARLY_AP_STARTUP by default in 12 and possibly 11.x).

2. Is the underlying issue that you don't handle MSI-X interrupt migration correctly? If so, we have a workaround for that case for Xen that you could use which prevents all migration (not just the boot-time migration).

• On Hyper-V MSI-Xs are always allocated after AP started with the WIP Hyper-V PCI-bridge code for PCI passthrough/SR-IOV, so re-shuffle is actually not involved.
• On Hyper-V, only atkbd and ata allocate interrupts from PIC (let's ignore the fdc there :), which are not performance critical at all.

The main issue, we found is that when re-shuffle happened, CAM was issuing commands to ata controller, and the ata interrupt was lost due to the reshuffle, which leaded to all kinds of weirdness later on (mainly because the disk itself is shared w/ the ata controller and the Hyper-V synthetic controller).

So the odd thing there is that in theory the shuffle shouldn't happen during CAM device discovery. The normal device discovery happens during the interrupt hooks run at SI_SUB_INT_CONFIG_HOOKS and thread0 waits for any interrupt hooks to finish before proceeding. The CAM bus scans only clear their hook once the bus scan is finished. However, what may be happening is that GEOM decides to asynchronously kick off tasting for the various partition providers, etc. after the CAM bus scans have finished. Even then, interrupts still shouldn't be lost as interrupt migration for an I/O APIC pin is careful to first mask the pin, then update it with the new (CPU, IDT) pair. After that is done, the old (CPU, IDT) pair is released via apic_free_vector() which uses sched_bind() to move to the old CPU and leave interrupts disabled for at least one instruction to catch any "in-flight" interrupts on the old CPU.

Certain commands are sent after disks are attached asynchronously after the config_hook, AFAIR, writecache and readahead etc.

It might be worth adding a 'DELAY(100)' or so after the sched_bind() to give any in-flight interrupts more time to hit the destination CPU (though the sched_bind() is going to go through an IPI send/ack if we weren't already running on the old CPU).

I am not going to mess up CAM too much this time, since as you have noted EARLY_AP will be enabled by default on 11 and 12. So I believe disabling the reshuffle on Hyper-V is the best practice (well, I also need to MFC it to 10).

So my concern is that if the I/O APIC emulation in Hyper-V doesn't handle shuffling, then that means that users using 'cpuset -x' to move interrupts around from the command line can also lose interrupts. In that case, I think we should disable interrupt migration on Hyper-V in general (not just in the shuffle). However, it would be helpful to know what is actually broken. Xen had a bug with MSI-X in particular such that older versions of Xen didn't handle writes to MSI-X table entries while MSI-X was enabled (but the entry was masked). That affected 'cpuset -x' as well as the boot round-robin, so for Xen we disable all MSI-X migration on hypervisors with the bug.

I will check Dexuan about MSI-X migration, since PCI passthrough is kinda working (we get a working passthrough-ed ix w/ MSI-X). However, if I recall correctly, he told me MSI-X migration works w/ 'cpuset -x' (not under heavy load through). So currently I believe we only need to prevent line based interrupts migration, which really goes through IOAPIC.

Can cause deadlock. It will be fixed in the upcoming channel dynamic detach/attach work.

In D7914#165285, @jhb wrote:

In D7914#165153, @sepherosa_gmail.com wrote:

In D7914#165049, @jhb wrote:

In theory the callers should be doing this (e.g. during something like resume where we write all the BARs). In addition, the ROM BAR shouldn't toggle MEMEN for all other memory BARs since it has its own bit. I'd rather we fix the callers that need this changed. There are some places in pci.c that aren't quite correct. Are you using this in out-of-tree changes or are you having issues with the existing uses (e.g. during boot-time probe)?

It's for WIP Hyper-V PCI bridge for PCI passthrough/SR-IOV. The WIP bridge driver will program the BARs when its children try to allocate RES_MEMORY, however, if the MEMEN is set (e.g. Upon the first RES_MEMORY request w/ RF_ACTIVE, the MEMEN will be set. When second RES_MEMORY request is received, normally for MSI-X, the MEMEN has been set), the hypervisor will ignore the programmed value, the exact path is:
pci_alloc_resource -> pci_reserve_map -> pci_write_bar.

Do you mean we should move the MEMEN clear/set to pci_reserve_map?

Yes. Truth be told, what we really should be doing is not setting MEMEN or IOEN unless/until all BARs of a given resource type are programmed/valid. I haven't gotten around to fixing that. For now, doing the disable/enable in pci_reserve_map() should be ok.

In D7949#165286, @jhb wrote:

In D7949#165151, @sepherosa_gmail.com wrote:

In D7949#165052, @jhb wrote:

A couple of comments:

1. The shuffle is going to go away (I plan to enable EARLY_AP_STARTUP by default in 12 and possibly 11.x).

2. Is the underlying issue that you don't handle MSI-X interrupt migration correctly? If so, we have a workaround for that case for Xen that you could use which prevents all migration (not just the boot-time migration).

• On Hyper-V MSI-Xs are always allocated after AP started with the WIP Hyper-V PCI-bridge code for PCI passthrough/SR-IOV, so re-shuffle is actually not involved.
• On Hyper-V, only atkbd and ata allocate interrupts from PIC (let's ignore the fdc there :), which are not performance critical at all.

The main issue, we found is that when re-shuffle happened, CAM was issuing commands to ata controller, and the ata interrupt was lost due to the reshuffle, which leaded to all kinds of weirdness later on (mainly because the disk itself is shared w/ the ata controller and the Hyper-V synthetic controller).

So the odd thing there is that in theory the shuffle shouldn't happen during CAM device discovery. The normal device discovery happens during the interrupt hooks run at SI_SUB_INT_CONFIG_HOOKS and thread0 waits for any interrupt hooks to finish before proceeding. The CAM bus scans only clear their hook once the bus scan is finished. However, what may be happening is that GEOM decides to asynchronously kick off tasting for the various partition providers, etc. after the CAM bus scans have finished. Even then, interrupts still shouldn't be lost as interrupt migration for an I/O APIC pin is careful to first mask the pin, then update it with the new (CPU, IDT) pair. After that is done, the old (CPU, IDT) pair is released via apic_free_vector() which uses sched_bind() to move to the old CPU and leave interrupts disabled for at least one instruction to catch any "in-flight" interrupts on the old CPU.

In D7914#165049, @jhb wrote:

In theory the callers should be doing this (e.g. during something like resume where we write all the BARs). In addition, the ROM BAR shouldn't toggle MEMEN for all other memory BARs since it has its own bit. I'd rather we fix the callers that need this changed. There are some places in pci.c that aren't quite correct. Are you using this in out-of-tree changes or are you having issues with the existing uses (e.g. during boot-time probe)?

In D7949#165052, @jhb wrote:

A couple of comments:

1. The shuffle is going to go away (I plan to enable EARLY_AP_STARTUP by default in 12 and possibly 11.x).

2. Is the underlying issue that you don't handle MSI-X interrupt migration correctly? If so, we have a workaround for that case for Xen that you could use which prevents all migration (not just the boot-time migration).

Most parts no longer apply; and we will take different approach, i.e. vmbus_xact.

No longer applies; and I don't think we would need it.