Page MenuHomeFreeBSD

Fix queue distribution when there are no threads
ClosedPublic

Authored by shurd on Apr 17 2018, 7:16 PM.
Tags
None
Referenced Files
Unknown Object (File)
Dec 20 2023, 2:30 AM
Unknown Object (File)
Aug 18 2023, 10:56 PM
Unknown Object (File)
Jun 10 2023, 2:38 PM
Unknown Object (File)
May 27 2023, 4:28 AM
Unknown Object (File)
May 7 2023, 2:02 AM
Unknown Object (File)
Jan 8 2023, 2:34 AM
F3316837: graph.png
Apr 18 2018, 9:09 AM
F3316188: graph.png
Apr 17 2018, 11:04 PM
Subscribers

Details

Summary

Previously, if there are no threads, all queues which targeted
cores that share an L2 cache were bound to a single core. The intent is
to distribute them across these cores.

Test Plan

Test on systems with shared L2 cache and no threads

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Panic on PC engines APU2 (AMD GX-412TC):

igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci1
igb0: attach_pre capping queues at 4
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 4
igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb0: using 4 rx queues 4 tx queues
igb0: Using MSIX interrupts with 5 vectors
igb0: allocated for 4 tx_queues
igb0: allocated for 4 rx_queues
taskqgroup_attach_cpu: qid not found for rxq2 cpu=-1
igb0: taskqgroup_attach_cpu failed 22
igb0: Failed to allocate que int 2 err: 22igb0: IFDI_MSIX_INTR_ASSIGN failed 22
panic: resource_list_release: can't find resource
cpuid = 3
time = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff8253a0c0
vpanic() at vpanic+0x18d/frame 0xffffffff8253a120
panic() at panic+0x43/frame 0xffffffff8253a180
resource_list_release() at resource_list_release+0x1b8/frame 0xffffffff8253a1c0
em_free_pci_resources() at em_free_pci_resources+0x6b/frame 0xffffffff8253a200
em_if_detach() at em_if_detach+0xb9/frame 0xffffffff8253a240
iflib_device_register() at iflib_device_register+0x211f/frame 0xffffffff8253a590
iflib_device_attach() at iflib_device_attach+0xb7/frame 0xffffffff8253a5c0
device_attach() at device_attach+0x3f6/frame 0xffffffff8253a610
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253a640
pci_attach() at pci_attach+0xd5/frame 0xffffffff8253a680
device_attach() at device_attach+0x3f6/frame 0xffffffff8253a6d0
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253a700
acpi_pcib_pci_attach() at acpi_pcib_pci_attach+0xa1/frame 0xffffffff8253a740
device_attach() at device_attach+0x3f6/frame 0xffffffff8253a790
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253a7c0
pci_attach() at pci_attach+0xd5/frame 0xffffffff8253a800
device_attach() at device_attach+0x3f6/frame 0xffffffff8253a850
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253a880
acpi_pcib_acpi_attach() at acpi_pcib_acpi_attach+0x450/frame 0xffffffff8253a8f0
device_attach() at device_attach+0x3f6/frame 0xffffffff8253a940
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253a970
acpi_attach() at acpi_attach+0xe49/frame 0xffffffff8253aa10
device_attach() at device_attach+0x3f6/frame 0xffffffff8253aa60
bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8253aa90
device_attach() at device_attach+0x3f6/frame 0xffffffff8253aae0
bus_generic_new_pass() at bus_generic_new_pass+0x118/frame 0xffffffff8253ab10
root_bus_configure() at root_bus_configure+0x77/frame 0xffffffff8253ab40
configure() at configure+0x9/frame 0xffffffff8253ab50
mi_startup() at mi_startup+0x118/frame 0xffffffff8253ab70
btext() at btext+0x2c
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
db>

And same on Intel Atom C2558:

igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3
igb0: attach_pre capping queues at 2
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 2
igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb0: using 2 rx queues 2 tx queues
igb0: Using MSIX interrupts with 3 vectors
igb0: allocated for 2 tx_queues
igb0: allocated for 2 rx_queues
taskqgroup_attach_cpu: qid not found for rxq1 cpu=-1
igb0: taskqgroup_attach_cpu failed 22
igb0: Failed to allocate que int 1 err: 22igb0: IFDI_MSIX_INTR_ASSIGN failed 22

Re-copy the CPU mask before selecting the core.

Lot's better on 4core AMD, but need to reset my non-IPMI Netgate tomorrow on the lab:

graph.png (1×1 px, 76 KB)

Same improvement on 8-cores ATOM with ixgbe drivers:

graph.png (1×1 px, 90 KB)

This revision is now accepted and ready to land.Apr 18 2018, 3:22 PM
This revision was automatically updated to reflect the committed changes.