Details

Reviewers

sbruno
shurd
kbowling
gallatin
jeffrey.e.pieper_intel.com

Group Reviewers

Intel Networking

Commits

rS339338: ixl/iavf(4): Change ixlv to iavf and update it to use iflib(9)

Summary

(Summary forthcoming)

A list of git commits that are contained in this revision can be found here: https://github.com/freebsd/freebsd/compare/master...intel-wired-ethernet:ixl-iflib-sriov

Also includes changes to the PF driver (ixl(4)).

Includes the contents of D16428 already.

Test Plan

(To be filled out)

At least run the VF driver with an ixl(4) host bare metal and using bhyve, and a Linux (RHEL?) host using KVM.

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

ixl(4): Fix module Makefile when PCI_IOV is enabled

Harbormaster completed remote builds in B18360: Diff 45891.Jul 26 2018, 9:08 PM

In general, I looked around at ixlv.c, and it looks like there are many functions for the main pf which are identical, or nearly identical for the VF. Can we please work on collapsing them down to common code?

sys/dev/ixl/ixl_pf_main.c
2364 ↗	(On Diff #45891)	I just hate that there are 2 functions which do almost the same thing. Can't you just change the args to ixl_disable_rings() to take pf and qtag and do this in a single function?
sys/dev/ixl/ixl_txrx.c
501 ↗	(On Diff #45891)	Errant space.

ixl(4): Combine PF-only/VF-only ixl_disable_rings() in PF driver into one function.
Remove a space

Harbormaster completed remote builds in B18537: Diff 46214.Aug 2 2018, 8:47 PM

@gallatin, I address your specific comments, but I would need to look through ixlvc.c some more to find more code that can be shared. There is almost certainly some in there that can be shared between the PF/VF drivers.

ixl(4): Lie to the VF about configuring promiscuous mode if VF isn't allowed to configure it

Harbormaster completed remote builds in B18564: Diff 46257.Aug 3 2018, 6:33 PM

Changes from D16428:

iflib: Prevent kernel panic on detach in _task_fn_admin by checking if in detach
iflib: Add STATE_LOCK() around flag setting in iflib_request_reset()

New changes for D16429:

ixl(4): Add tunable to enable VF loopback; enable by default
ixl/ixlv(4): Fix dev.ixl/v.#.vsi.rxq#.bytes sysctl not incrementing

Harbormaster completed remote builds in B18682: Diff 46482.Aug 9 2018, 6:40 PM

ixl(4): Fix compile error when PCI_IOV isn't defined

Harbormaster completed remote builds in B18688: Diff 46490.Aug 9 2018, 10:24 PM

gallatin accepted this revision.Aug 10 2018, 3:41 PM

This revision is now accepted and ready to land.Aug 10 2018, 3:41 PM

Several outstanding issues:

System back trace while running netperf tcp_rr test with jumbo frames enabled:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x1af0000
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80bdeb4f
stack pointer = 0x28:0xfffffe0025eeccb0
frame pointer = 0x28:0xfffffe0025eeccc0
code segment = base 0x0, limit 0xfffff, type 0x1b

= DPL 0, pres 1, long 1, def32 0, gran 1

processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))
[ thread pid 12 tid 100034 ]
Stopped at m_tag_delete_chain+0x1f: movq (%rsi),%rdi
db> bt
Tracing pid 12 tid 100034 td 0xfffff800032c4580
m_tag_delete_chain() at m_tag_delete_chain+0x1f/frame 0xfffffe0025eeccc0
uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0025eecd20
m_freem() at m_freem+0x9a/frame 0xfffffe0025eecd40
vtnet_txq_eof() at vtnet_txq_eof+0x75/frame 0xfffffe0025eecd70
vtnet_txq_mq_start_locked() at vtnet_txq_mq_start_locked+0x65/frame 0xfffffe0025eecdd0
vtnet_txq_mq_start() at vtnet_txq_mq_start+0x68/frame 0xfffffe0025eece10
ether_output_frame() at ether_output_frame+0xaa/frame 0xfffffe0025eece40
ether_output() at ether_output+0x6b7/frame 0xfffffe0025eecee0
ip_output() at ip_output+0x13e6/frame 0xfffffe0025eed020
tcp_output() at tcp_output+0x1b0c/frame 0xfffffe0025eed1b0
tcp_timer_delack() at tcp_timer_delack+0xf4/frame 0xfffffe0025eed1f0
softclock_call_cc() at softclock_call_cc+0x12e/frame 0xfffffe0025eed2a0
softclock() at softclock+0x79/frame 0xfffffe0025eed2c0
ithread_loop() at ithread_loop+0x1a7/frame 0xfffffe0025eed330
fork_exit() at fork_exit+0x83/frame 0xfffffe0025eed370
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0025eed370

trap 0, rip = 0, rsp = 0, rbp = 0 ---

db>

Fatal trap 9: general protection fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer = 0x20:0xffffffff82d70048
stack pointer = 0x28:0xfffffe0025ec5150
frame pointer = 0x28:0xfffffe0025ec5180
code segment = base 0x0, limit 0xfffff, type 0x1b

= DPL 0, pres 1, long 1, def32 0, gran 1

processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (if_io_tqg_1)
[ thread pid 0 tid 100026 ]
Stopped at ixl_isc_rxd_pkt_get+0xc8: cmpl ll+0x1b(%rax),%esi
db> bt
Tracing pid 0 tid 100026 td 0xfffff800032ad580
ixl_isc_rxd_pkt_get() at ixl_isc_rxd_pkt_get+0xc8/frame 0xfffffe0025ec5180
iflib_rxeof() at iflib_rxeof+0x136/frame 0xfffffe0025ec5260
_task_fn_rx() at _task_fn_rx+0x76/frame 0xfffffe0025ec52a0
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x144/frame 0xfffffe0025ec5300
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x98/frame 0xfffffe0025ec5330
fork_exit() at fork_exit+0x83/frame 0xfffffe0025ec5370
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0025ec5370

trap 0, rip = 0, rsp = 0, rbp = 0 ---

db>

It is possible to work around the issue by increasing the value of the rx_budget counter to the current value of the number of queues (44):
sysctl dev.ixl.0.iflib.rx_budget=44

Unable to initialize over 63 VF's:

Aug 10 14:48:56 u1017 kernel: ixl3: 0x400000 bytes of rid 0x184 res 3 failed (0, 0xffffffffffffffff).

Memory leak reported when unloading if_ixlv with 64 VF’s spawned:

Aug 10 14:51:18 u1017 kernel: ixlv62: Using MSIX interrupts with 5 vectors
Aug 10 14:51:18 u1017 kernel: ixlv62: Ethernet address: e2:db:49:42:7a:85
Aug 10 14:51:18 u1017 kernel: ixlv62: netmap queues/slots: TX 4/1024, RX 4/1024
Aug 10 14:51:18 u1017 kernel: ixlv63: <Intel(R) Ethernet Virtual Function 700 Series (X722)> at device 0.79 on pci13
Aug 10 14:51:18 u1017 kernel: ixlv62: link state changed to UP
Aug 10 14:51:18 u1017 kernel: ixlv63: PF API 1.1 / VF API 1.1
Aug 10 14:51:18 u1017 kernel: ixlv63: VSIs 1, QPs 4, MSIX 5, RSS sizes: key 52 lut 64
Aug 10 14:51:18 u1017 kernel: ixlv63: using 1024 tx descriptors and 1024 rx descriptors
Aug 10 14:51:18 u1017 kernel: ixlv63: msix_init qsets capped at 4
Aug 10 14:51:18 u1017 kernel: ixlv63: pxm cpus: 36 queue msgs: 4 admincnt: 1
Aug 10 14:51:18 u1017 kernel: ixlv63: using 4 rx queues 4 tx queues
Aug 10 14:51:18 u1017 kernel: ixlv63: Using MSIX interrupts with 4 vectors
Aug 10 14:51:18 u1017 kernel: ixlv63: failed to allocate IRQ for rid 5, name rxq3.
Aug 10 14:51:18 u1017 kernel: ixlv63: _iflib_irq_alloc failed 12
Aug 10 14:51:18 u1017 kernel: ixlv63: Failed to allocate queue RX int vector 3, err: 12
Aug 10 14:51:18 u1017 kernel: ixlv63: IFDI_MSIX_INTR_ASSIGN failed 12
Aug 10 14:51:18 u1017 kernel: device_attach: ixlv63 attach returned 12
Aug 10 14:51:18 u1017 kernel: pci0:132:0:79: Device leaked MSI vectors
Aug 10 14:51:18 u1017 kernel: pci0:132:0:79: Device leaked memory resources

After kldunload:

Aug 10 15:00:18 u1017 kernel: ixlv62: detached
Aug 10 15:00:18 u1017 kernel: pci13: <network, ethernet> at device 0.78 (no driver attached)
Aug 10 15:00:18 u1017 kernel: Warning: memory type ixlv leaked memory on destroy (4 allocations, 8192 bytes leaked).

This revision now requires changes to proceed.Aug 10 2018, 4:20 PM

ixl/ixlv(4): Fix panic when receiving jumbo frame (>4k mtu) traffic

Harbormaster completed remote builds in B18703: Diff 46521.Aug 10 2018, 5:37 PM

ixlv(4): Change remaining M_DEVBUF malloc types to M_IXLV
ixlv(4): Fix a memory leak when interrupt setup fails
ixlv(4): Convert some malloc()'s to M_WAITOK from M_NOWAIT
ixlv(4): Remove comments and commented out code
ixlv(4): Remove list of media types displayed in "ifconfig -m"

Harbormaster completed remote builds in B18780: Diff 46642.Aug 14 2018, 12:03 AM

ixl(4): Enhance Malicious Driver Detection messages with VF information
ixl(4): Print out additional event information when a LAN Overflow Event occurs
ixl/ixlv(4): Add additional MPASS() asserts to the driver TX path

Harbormaster completed remote builds in B18879: Diff 46809.Aug 16 2018, 11:45 PM

gallatin added inline comments.Aug 17 2018, 7:34 PM

sys/dev/ixl/if_ixl.c
1199 ↗	(On Diff #46809)	Please consider limiting this to a few prints. Otherwise, I anticipate this spamming the console / messages log at some point for somebody.
sys/dev/ixl/ixl_pf_main.c
2426 ↗	(On Diff #46809)	Nice! This would have been helpful during bringup of the iflib driver, but better late than never :)
sys/dev/ixl/ixl_txrx.c
364 ↗	(On Diff #46809)	0 is a perfectly valid DMA address on non-x86, or on x86 when using IOMMUs

ixl(4): Fix RX MDD event display
ixl/ixlv(4): Add more MPASS statements to TX path

Harbormaster completed remote builds in B18994: Diff 47009.Aug 21 2018, 12:42 AM

ixlv(4): Make ixlv_if_init() and ixlv_stop() wait for ENABLE/DISABLE queue command completion
ixl(4): Cosmetic change to message PF sends in reply to VF promiscuous config message

Harbormaster completed remote builds in B19027: Diff 47074.Aug 22 2018, 12:48 AM

ixl/ixlv(4): Fix TXD credits update for DWB mode

Harbormaster completed remote builds in B19111: Diff 47190.Aug 23 2018, 5:56 PM

ixl(4): Fix memory leak when head writeback is disabled

Harbormaster completed remote builds in B19117: Diff 47209.Aug 23 2018, 9:22 PM

ixl/ixlv Missing semicolon in DWB fix

Harbormaster completed remote builds in B19214: Diff 47347.Aug 27 2018, 3:43 PM

iflib: Destroy state lock at end of iflib_device_deregister()
iflib: Move ifc_in_detach variable to flag in if_ctx_t's ifc_flags, and lock around it
ixl(4): Fix kernel panic during init() when Admin Queue is not alive

Harbormaster completed remote builds in B19307: Diff 47509.Aug 31 2018, 12:01 AM

ixlv(4): Swap MALLOC_DEFINE/DECLARE statements
ixlv(4): Remove commented out code
ixlv(4): Check return value of ixlv_add_mac_filter when adding MC addresses
ixlv(4): Update some debug printf()'s to use newer debug functions
ixlv(4): malloc() the correct size for the heads of the MAC/vlan filter lists

Harbormaster completed remote builds in B19398: Diff 47671.Sep 4 2018, 11:33 PM

ixlv(4): Disable queues before initialization.

Harbormaster completed remote builds in B19573: Diff 48012.Sep 13 2018, 6:00 PM

ixl(4): Configure promiscuous mode during driver init

Harbormaster completed remote builds in B19583: Diff 48025.Sep 13 2018, 11:00 PM

ixl(4): Fix compile error introduced by previous commit

Harbormaster completed remote builds in B19585: Diff 48027.Sep 13 2018, 11:10 PM

ixl(4): Fix filter handling when SR-IOV is enabled
Revert "iflib: Free RX queue TCP LRO memory on unload"

Harbormaster completed remote builds in B19703: Diff 48274.Sep 20 2018, 5:40 PM

ixlv(4): Update driver version to 2.0.0-k
ixlv(4): Remove dynamic ITR code, since it's unused
ixlv(4): Set max number of queues used in iflib based on number VF receives from PF
ixlv(4): Code cleanup
ixlv(4): Don't sleep for a response to a VC msg if driver is in detach
ixl(4): Clear VF qtag queue flags when VF is being reinitialized
ixlv(4): Move queue disable to earlier in ixlv_if_init, and don't sleep for completion
ixlv(4): Modify debug prints, and stop printing out "Queue already enabled" warnings
ixl(4): Update version number to 2.1.0-k

Harbormaster completed remote builds in B19831: Diff 48493.Sep 27 2018, 12:56 AM

erj mentioned this in D15927: Update ixl(4) and ixlv(4) man pages.Oct 2 2018, 3:54 PM

Rebased onto latest master, and added:

ixl(4): Reconfigure MAC/VLAN filters after adding VEB during SR-IOV init

Harbormaster completed remote builds in B19934: Diff 48688.Oct 3 2018, 8:10 PM

Ugh, rebasing changes the commit SHAs :(

I guess it's the price you pay for the appearance of cleanliness...

kgalazka mentioned this in D17404: iflib: drain enqueued tasks before detaching from taskqgroup.Oct 4 2018, 12:32 PM

ixlv/iavf(4): Change "ixlv" driver name to "iavf"

Harbormaster completed remote builds in B19990: Diff 48761.Oct 4 2018, 10:29 PM

erj retitled this revision from ixlv(4): Update to use iflib to ixlv(4): Update to use iflib; change name to iavf(4).Oct 4 2018, 10:31 PM

iavf(4): Move modules/Makefile entry so that it's in alphabetical order
iavf(4): Update MODULE_* macros with new "iavf" name

Harbormaster completed remote builds in B20010: Diff 48795.Oct 5 2018, 6:52 PM

erj added a parent revision: D16428: iflib: Changes for ixlv(4) update.Oct 5 2018, 8:21 PM

erj added a child revision: D15927: Update ixl(4) and ixlv(4) man pages.

Can you rename the sys/dev/ixl dir to iavf as part of this? I'd really like to see this change done now in 12.0, it would be very awkward to try and MFC later.

Sorry the first part of my comment made no sense, ixl is the pf and will remain for 25/40g hw parts.

jeffrey.e.pieper_intel.com accepted this revision.Oct 11 2018, 8:14 PM

This revision is now accepted and ready to land.Oct 11 2018, 8:14 PM

sbruno accepted this revision.Oct 11 2018, 10:48 PM

Any last-minute assents or objections?

kbowling accepted this revision.Oct 12 2018, 5:55 AM

erj removed a parent revision: D16428: iflib: Changes for ixlv(4) update.Oct 12 2018, 5:28 PM

Closed by commit rS339338: ixl/iavf(4): Change ixlv to iavf and update it to use iflib(9) (authored by erj). · Explain WhyOct 12 2018, 10:41 PM

This revision was automatically updated to reflect the committed changes.

erj mentioned this in rS339634: iflib: drain enqueued tasks before detaching from taskqgroup.Oct 23 2018, 4:37 AM

ixlv(4): Update to use iflib; change name to iavf(4)
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 49079

head/sys/amd64/conf/GENERIC

head/sys/amd64/conf/NOTES

head/sys/conf/files.amd64

head/sys/dev/ixl/README

head/sys/dev/ixl/i40e_osdep.c

head/sys/dev/ixl/if_ixl.c

head/sys/dev/ixl/if_ixlv.c

head/sys/dev/ixl/ixl.h

head/sys/dev/ixl/ixl_debug.h

head/sys/dev/ixl/ixl_pf.h

head/sys/dev/ixl/ixl_pf_iov.h

head/sys/dev/ixl/ixl_pf_iov.c

head/sys/dev/ixl/ixl_pf_main.c

head/sys/dev/ixl/ixl_pf_qmgr.h

head/sys/dev/ixl/ixl_pf_qmgr.c

head/sys/dev/ixl/ixl_txrx.c

head/sys/dev/ixl/ixlv.h

head/sys/dev/ixl/ixlv_vc_mgr.h

head/sys/dev/ixl/ixlvc.c

head/sys/modules/Makefile

head/sys/modules/iavf/Makefile

head/sys/modules/ixl/Makefile

head/sys/modules/ixlv/Makefile

head/sys/net/iflib.h

head/sys/net/iflib.c

head/sys/net/iflib_private.h

ixlv(4): Update to use iflib; change name to iavf(4)ClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 49079

head/sys/amd64/conf/GENERIC

head/sys/amd64/conf/NOTES

head/sys/conf/files.amd64

head/sys/dev/ixl/README

head/sys/dev/ixl/i40e_osdep.c

head/sys/dev/ixl/if_ixl.c

head/sys/dev/ixl/if_ixlv.c

head/sys/dev/ixl/ixl.h

head/sys/dev/ixl/ixl_debug.h

head/sys/dev/ixl/ixl_pf.h

head/sys/dev/ixl/ixl_pf_iov.h

head/sys/dev/ixl/ixl_pf_iov.c

head/sys/dev/ixl/ixl_pf_main.c

head/sys/dev/ixl/ixl_pf_qmgr.h

head/sys/dev/ixl/ixl_pf_qmgr.c

head/sys/dev/ixl/ixl_txrx.c

head/sys/dev/ixl/ixlv.h

head/sys/dev/ixl/ixlv_vc_mgr.h

head/sys/dev/ixl/ixlvc.c

head/sys/modules/Makefile

head/sys/modules/iavf/Makefile

head/sys/modules/ixl/Makefile

head/sys/modules/ixlv/Makefile

head/sys/net/iflib.h

head/sys/net/iflib.c

head/sys/net/iflib_private.h

ixlv(4): Update to use iflib; change name to iavf(4)
ClosedPublic
Actions

Revision Contents
Changeset List