Initially, RSF (Receive Queue Store and Forward) was disabled for
unknown reasons, but the cut-through mode that's enabled as a result
seems to send 0 length packets up to the DMA when the RX queue is
full.
I'll close the relevant bug report as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252850
Looks good to me.
Tested and confirmed to be working on the AMD SoC 10GbE NIC. Right now this is the only way to get a consistent link-up during boot and when hotplugging SFP+ modules. The internal receiver reset cycle might contain a hardware issue that prevents it from working properly.
Actually, the sysctl's are already interface specific only. User can either set the global "kenv" dev.ax.sph_enable to configure sph on both interface. Or they can use the interface specific "sysctl" dev.ax.X.sph_enabled to configure the same specific to interface.
For now, as mentioned earlier, modified this patch to be driver specific and get netmap bridging works with the driver. Please let me know if any comments.
@rajesh1.kumar_amd.com Also, the sysctl's need to be changed to be interface-specific if we are deciding to use the "dev.ax.X.sph_enable" format for increased flexibility. See https://reviews.freebsd.org/D27800
@rajesh1.kumar_amd.com Thanks for the patch. It should be noted that disabling the split header function now requires the single_fl to be set. If that's acceptable, I suggest updating the manpage to explicitly document this, as right now it's documented as "recommended".
In D27797#625707, @vmaffione wrote:Are both netmap_test and sph_enable actually necessary? Would it make sense to have only sph_enable, and set isc_nfl = 1 when sph_enable = 0 and isc_nfl = 2 when sph_enable = 1?
If not, I think netmap_test should be called something like single_free_list
If you look at vmxnet3_isc_rxd_refill() it looks like freelist #0 (and rxq
#0) is for BTYPE_HEAD descriptors, whereas freelist #1 (and rxq#1) is for
BTYPE_BODY, for each qset. This seems to point at a sort of split
descriptor strategy. However I don't have the vmxnet3 specifications, so I
don't know for sure.To be honest I'm quite a bit confused about what combinations of isc_nfl
and isc_nrxqs are valid, and what their meaning is in general, except for
the simple case where isc_nfl == isc_nrxqs == 1. It looks like drivers
where isc_nrxqs > 1 have one completion ring plus 1 or 2 command rings..
But this is something that should depend on the NIC specifications. So I
don't understand how you can "decide" to go for isc_nfl == 2 && isc_nrxqs1 rather than isc_nfl == 2 && isc_nrxqs == 2. If you have a better
understanding than mine please do let me know.
I should mention that this change also includes two bugfixes:
Hi Rajesh,
In my setup, port_mode is SFP, and sfp_base is XGBE_SFP_BASE_10000_SR, where auto-negotiation is disabled. But I still see the same behavior. From earlier logs, your port mode is also SFP, but not sure about sfp_base and whether Auto-negotiation is enabled. Can you please check that (enabling verbose logs during hotplug would give that info)? It would be good if you can share a verbose log capture from your setup (starting from driver load).
Of course, here are the relevant hotplug logs:
Hi Rajesh,
Regarding the port0 - port1 direct connection back-to-back. I just did a quick test in my setup here. I couldn't see the toggling issue. As expected, both side link goes down when I pull the cable/SFP module just one side and coming up. But thanks for bringing this, let me do some more testing to see the problem.
Keep in mind that this test was performed using a fiber connection. Using an SFP module with a copper cable (RJ45) produces a different result: only one port establishes a link, the other port does not, this does not seem to change over time.
Hello Rajesh,
Regarding the link issue you are facing, can you try setting "an_cdr_workaround = 0" in if_axgbe_pci.c in appropriate version and give a try?
Unfortunately, this does not seem to fix the issue.
Hi Rajesh,
Unfortunately, I couldn't do the hotplug test until Oct 5th with my remote setup as mentioned before. Sorry about that. I will do my hotplug testing next week and see whether I can recreate the problem. Until then, I will see if I can recreate the problem somehow.
That's no problem at all. The link issue is indeed hotplug-related. I will also continue to see if I can recreate the problem. I suspect that it is a timing issue, so I'm logging everything to see if I can find the issue.
Hi Rajesh, apologies for the late reply.
Just to make sure on my understanding. By link issue, you just mean the delay of more than 30 secs for Link UP? Or delayed link up + link not coming up intermittently? Just wanted to make sure whether you see link not coming up issue after you fix the EEPROM read issue.
I mean the delayed link up + link not coming up intermittently.
So, with the increased timeout I believe you are not seeing EERPOM read issue and link always coming up. But, are you still facing the delays in link up often? Is it blocking your progress by anyway?
The increased timeout indeed eliminates the issue of reading the EEPROM. I don't think this has anything to do with the link issue, as there is (as far as I know) a service timer that executes every second to see if any signals have changed, it then proceeds to read the EEPROM again, regardless of whether the module has changed. We should not see a Link UP after 30 seconds or more (or never) if the service timer executes every second. Could this have something to do with the integration with iflib? My suspicion comes from the fact that the "Link UP" message comes from iflib.
By the way, I also applied the logic analyzer to the Linux driver and saw the exact same results, except no EEPROM read fail....
Regarding the following statement, Does it mean the Link doesn't come up at all? or It takes some delay to come up?
Frequently, the link does not seem to be established (or at least, not right away).
Hi Rajesh,