Verified in NetApp lab.

In D30733#704658, @krzysztof.galazka_intel.com wrote:

In D30733#701137, @stallamr_netapp.com wrote:

e2a: ixl_link_event Status_LN: 0xca, HW_LN: 0xca, Status_AN: 0x0, HW_AN: 0x80 <<<<< spurious event
e2a: ixl_link_event IFF_UP: 1 MA: 64, LUP: 0 QUAL: 0 <<< spurious event
e2a: ixl_link_event Status_LN: 0xe1, HW_LN: 0xe1, Status_AN: 0x80, HW_AN: 0x80
e2a: ixl_link_event IFF_UP: 1 MA: 64, LUP: 1 QUAL: 128

I don't think we can do much about those spurious events in the driver, but that AN status retrieved with Get Link Status AQC has correct information. Using It instead of information from the event should help.

In D30733#700207, @krzysztof.galazka_intel.com wrote:

I'm still not able to reproduce this issue with any of my switches. Could you, please, modify printf in the ixl_link_event function to dump hex values of status->link_info, status->an_info, hw->phy.link_info.link_info and hw->phy.link_info.an_info, and send me the log?

In D30733#694491, @krzysztof.galazka_intel.com wrote:

In D30733#693043, @stallamr_netapp.com wrote:

I have connected the cable back-to-back between two servers and rebooted them both at a time. During boot, on both the nodes, immediately after the driver is attached, receives link-event and notices MEDIA_AVAILABLE + IFF_UP + UNQUALIFIED + NO_LINK_UP.

I'm testing using 4 port adapter with following config:

/boot/loader.conf:
dev.ixl.0.link_active_on_if_down=1
dev.ixl.3.link_active_on_if_down=0

/etc/rc.conf:
ifconfig_ixl0=190.2.20.1/16
ifconfig_ixl3=190.3.20.1/16

To ensure that link state is correct driver during attach sets it according to the link_active_on_if_down tunable. This triggers a link event but during attach IFF_UP flag is not set:
ixl3: ixl_set_link enable: 0
ixl3: ixl_link_event IFF_UP: 0 MA: 64, LUP: 0 QUAL: 128
ixl3: ixl_link_event IFF_UP: 0 MA: 64, LUP: 0 QUAL: 0

Then interface is brought up with an ioctl call:
ixl3: ixl_set_link enable: 1
ixl3: ixl_link_event IFF_UP: 1 MA: 64, LUP: 1 QUAL: 128
ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl3: link state changed to UP

With link_active_on_if_down=1 FW correctly reports that module is qualified in every link event:

ixl0: ixl_set_link enable: 1
ixl0: ixl_link_event IFF_UP: 0 MA: 64, LUP: 0 QUAL: 128
ixl0: ixl_link_event IFF_UP: 0 MA: 64, LUP: 1 QUAL: 128
ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl0: link state changed to UP

ixl0: ixl_set_link enable: 1
ixl0: ixl_link_event IFF_UP: 1 MA: 64, LUP: 0 QUAL: 128
ixl0: link state changed to DOWN
ixl0: ixl_link_event IFF_UP: 1 MA: 64, LUP: 1 QUAL: 128
ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl0: link state changed to UP

In D30889#694862, @erj wrote:

In D30889#694858, @stallamr_netapp.com wrote:

Eric,

I donot see "irdma" available yet in FreeBSD ?

Do you mean in-kernel? It's currently available as an OOT module here:

https://downloadcenter.intel.com/download/30325/irdma-RDMA-FreeBSD-Driver-for-Intel-Ethernet-Controller-E810

I donot see "irdma" available yet in FreeBSD ?

In D30733#692780, @krzysztof.galazka_intel.com wrote:

In D30733#691657, @stallamr_netapp.com wrote:

So, thinking on this, my guess is that when you reboot the machine, we would be finding an "unqualified" for a qualified cable because FW see this as link down. Also, my guess, the cable will show up as unqualified when you shut the link on link-partner.

I'm testing both scenarios with this transceiver:

plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: Intel Corp PN: AFBR-703SDZ-IN2 SN: AD1432A0AY9 DATE: 2014-08-11

and I don't see the unqualified message in the dmesg

Shutting down a link on the link partner does not affect reporting by FW if module is qualified.

I have applied the patch and rebooted my machine and I see "unqualified" message for a good cable.
The good thing is that with this patch I donot see "unqualified" message for a admin link-down.

Thanks Krzysztof , will hold-onto this review and wait on you for further direction.

So, thinking on this, my guess is that when you reboot the machine, we would be finding an "unqualified" for a qualified cable because FW see this as link down. Also, my guess, the cable will show up as unqualified when you shut the link on link-partner.

Thanks Krzysztof.

Also, in original change, https://reviews.freebsd.org/D28028, I noticed that after executing ixl_set_link(pf, false), the PHY capabilities query for an_info has I40E_AQ_QUALIFIED_MODULE unset. So, the same supported/qualified module becomes unqualified.
I think the crux of the problem is with ixl_set_link() unsetting I40E_AQ_QUALIFIED_MODULE.

Also, I prefer to have a quick call and discuss the ideas & thoughts we have. We would need an expert from Intel to help us understand AIM.
On side note, from NetApp performance experiments on NetApp platforms, BSD11 (legacy driver) vs BSD12 (IFLIB base driver) - we noticed almost 3.5x-4x latency spike in one of write tests for IFLIB-based drivers.

I have similar observation (bad news) wrt UDP. But for TCP, I see just fine. My runs are all on NetApp platform.
Please note: my client is not HoL.

Thanks Kevin. Your observations seem to fall inline with my code understanding. I have been digging into this to understand performance impact, NetApp is having after migrating to IFLIB based drivers. Here are my observations so far (mainly with respect to Intel 10G/40G drivers):

Thinking more on this, lets cancel/ignore this change. In other words, keep the default setting to FALSE. Anyone who wants AIM, can enable on their platforms.

Summarizing, this AIM will basically decide the value of ITR. Without AIM, ITR value will always stay at 128. With AIM enabled, ITR value now juggles between 32 - 8192. Below is the histogram on NetApp platform (12 processors, Intel Xeon D-1557 @1.50GHZ) when I had iperf3 ran for 1min.
I donot know exactly how ITR value translates to Interrupt moderation but to best of my knowledge, this value determines a minimum gap between two consecutive interrupts. From below histogram you can see, that, with more frequent data, ITR value concentrates more around 32 and 128.
That also implies, the gap between interrupts is lower when compared to AIM disabled i.e, 128. This falls inline with your observation too.

Review https://reviews.freebsd.org/D27465 helps to improve latency overall and has no dependency on this revision.

• Ensure to release MSIX after freeing IRQ resources

• Add IFDI_QUEUES_FREE() to iflib_pseudo_deregister()

Addressed Mark J comments.

• Move AIM functionality into its own function to keep interrupt handler code simple

This change with re-arm IRQ i.e, rx_wait_irq = 1, also helped NetApp to improve latency in 64K writes.

rx bitmap do get free in following function call. Remove explicit free of bitmap yet again.

Sure, thanks Kevin. I will discuss with Intel engineers.

At NetApp, in one of our proprietary application 64K sequential write load test, we noticed that average LRO size of the packet/segment gets shorter. And we also observed, average LRO segment size is proportional to interrupt rate and throttling. For better running of our application, we need average LRO segment size to be on higher size. With AIM enabled (and with other Intel patch), the average LRO size now come close to BSD11 LRO average segment size for same application.