One option maybe to split it into two rather than unified.

Other than the comment mentioned, this patch looks good to me.

In D21462#634278, @emaste wrote:

But somehow it's comparing ntb_hw_amd with ntb_transport always.

I think this is an artifact of git's copy/rename detection, and if that's the case there's some environment variable or cmd line option to disable it. I'll try to take a look for it and update the code review wiki page.

It can be merged there (MFC'd) as well, after a period of a minimum of 3 days. There is no need to submit a separate patch.

Thanks @vmaffione for reviewing and accepting the changes.

I take care of the commit and the MFC later on. Please submit a new differential with updates after the commit. It is so easier to review

@vmaffione, Sorry I just noticed that the stable/13 branch is created yesterday. So, can this patch be pushed to that branch as well? or do I need to submit the same patch separately for stable/13 branch?

@gbe, I see this manpage is still not in the main branch for FreeBSD-src. Can this be pushed to main branch and stable/13 branch?

Thanks @vmaffione for pushing it upstream.

In D28266#633046, @vmaffione wrote:

Will the fix be provided as a firmware upgrade or how?
I could commit this change if you don't have someone else around that can do that for you.

Any more comments on this patch?

In D28266#632505, @vmaffione wrote:

Wouldn't it be easier to always perform the complete phy reset if this is a known hardware issue? Or maye perform it automatically after a normal reset does not work?
I'm afraid that adding an obscure sysctl (obscure for anyone except for the axgbe developers) would be of little use. Unaware users wouldn't have a clue about what to do...
just my 2 cents

Thanks @gallatin for pushing the patch.

The kernel panic issue which I reported with pkt-gen here is root caused to an issue in the axgbe driver. Now that issue is fixed in the driver and I will submit the driver patches seperately.

@gallatin, I don't have the committer access for the git repo. Can you please help push this change upstream if there is no other comments?

Abandoning this revision and submitted another patch (D28266) which has this patch along with fixes for other known link issues with axgbe

Thanks @stephan.dewt_yahoo.co.uk for testing and giving your comments.

Thanks @vmaffione for committing the changes.

Shall I commit this change?

LGTM, since all previous comments are addressed.

I see "iflib" group review is blocking here. Now that this patch is only driver specific and doesn't have any change in iflib, does this patch need any other action to get it pushed upstream?

OK, thanks that makes sense. I was not considering a multi-frag case.

V6 patch addressing the last comment

V5 patch addressing all the previous comments.

If you tested all the valid combinations of sph_enable and netmap/non-netmap I don't have any objection in this being merged.

If there is no other comments, can this patch be accepted upstream?

V4 patch with a minor change to have both kenv and sysctl name to be same as
"sph_enable"

We faced this issue when testing the split header feature with "axgbe" driver.

When changing this via the sysctl interface (e.g. # sysctl dev.ax.0.sph_enabled=1), it returns with

sysctl: oid 'dev.ax.0.sph_enabled' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

Regarding the case sph_enable = 0 and single_fl = 0, I would like to add a
new isc_flag that you can set to tell iflib that you don't want to allow
netmap on this driver. In this way you can set the flag in case sph_enable
single_fl = 0, so that users won't do mistakes.

V3 patch with following changes

For now, I will better remove the "single_fl" sysctl and retain just the "sph_enable" and the changes (without any iflib changes) to get netmap bridging to work. Thanks @stephan.dewt_yahoo.co.uk for those changes. So, the next patch will be a driver only patch without any need for iflib changes.

V3 patch addressing previous comments.

V2 patch addressing the previous comments

Are both netmap_test and sph_enable actually necessary? Would it make sense to have only sph_enable, and set isc_nfl = 1 when sph_enable = 0 and isc_nfl = 2 when sph_enable = 1?
If not, I think netmap_test should be called something like single_free_list

I agree, both sysctl variables now depend on each other. I will create a change and submit it for review here once I find the time.

Ok, so the axgbe netmap receive datapath panics. Does this happen with or
without D27797?
If the panic is not introduced by your changes (e.g. if it is in HEAD) you
should submit a bug to the bugzilla.

Abandoning this revision as the necessary fix is done as part of D27980.

Done. We should therefore close this review.

You could then try something like:

pkt-gen -i ax0 -f tx -F 3

or

pkt-gen -i ax0 -f tx -F 2 -M 100

and check that the receiver indeed receives what you expect to receive.
Some useful pkt-gen options are -R (to rate limit the transmitter), and `-f
txseq` (on the sender) and -f rxseq (on the receiver) to let the
sender/receiver use sequence numbers and check if you lose packets.

@vmaffione, Thanks for the simplified patch. Network bridging with and without zerocopy is working properly (it needs D27799 netmap_txsync changes)

Thanks for your time and comments.

V2 patch with previous review comments addressed.

Thanks for the comments.

V2 patch with review comments addressed

Thanks @adrian for taking on this. From a compilation perspective, Looks good. I don't have any more specific comments.

@kevans, Is it possible to share your hardware details? do you have a test hardware with RJ45 ports? Any specific reason why it's been compiled with GCC?

GCC points out that this statement does nothing, XGBE_ADV expands to ((_phy)->advertising & ADVERTISED_##_mode)

Could I ask for a manpage too ?

Thanks @manu for committing this driver.

Talking to my peers here, looks like there is a similar issue being reported internally. That seems related to Auto-negotiation and PCS hardware. But, I am yet to get the full details of the same. I will update once I get the details.

I have been trying the hotplug/hotunplug experiment in my setup which is a back-to-back connection between the AMD SFP ports and an external link partner. As I have mentioned earlier, Intermittently I could see the delay in link coming up, and when that happens, I observe the similar behavior of PCS register having value of 0xC2 (even though sfp gpio signals are sensing the link immediately). We expect a value of 0x46 (bit[2] to be set basically). One possible reason could be the CDR workaround. But that option is ruled out. Adding to it, I see the delay only when the SFP module is touched, but not when just the link is hotplugged/unplugged. But I haven't experienced a scenario where the link doesn't come up at all.

Regarding the link issue you are facing, can you try setting "an_cdr_workaround = 0" in if_axgbe_pci.c in appropriate version and give a try?

Fixed the build issues with arm64 and corrected the config files as needed

Finally buildworld completes after around 3 hours. Looks like the issue is because of the missing files mentioned in files.<arch>

I am trying to build for arm64. It's taking a lot of time to buildworld (even with 16 threads). I will see how it goes.

Hi @manu. Fixed the GENERIC file. Please let me know if anything else needs to be taken care.

Fixed the sys/amd64/conf/GENERIC mis-merge

Fine @manu. I will update it.

Thanks for pointing the mis-merge @manu. Sorry about that.

Changes:

1. Increased timeout from 1 to 20 for SFP EEPROM read
2. Changes to add supports for appropriate media types
3. Changes to list appropriate media type in ifconfig output
4. Logging changes.

The issue is seen both during hotplug/hotunplug. As I said, I tried to create a reproducible scenario. During testing this morning, I tried:

The increased timeout indeed eliminates the issue of reading the EEPROM. I don't think this has anything to do with the link issue, as there is (as far as I know) a service timer that executes every second to see if any signals have changed, it then proceeds to read the EEPROM again, regardless of whether the module has changed. We should not see a Link UP after 30 seconds or more (or never) if the service timer executes every second.

Thanks @stephan.dewt_yahoo.co.uk for the detailed info and sharing the logs.

Regarding the following statement, Does it mean the Link doesn't come up at all? or It takes some delay to come up?

Fine. Good to hear you could progress @manu. I will make that logging change and update the patch.

It's a logging mistake @manu. I will correct it accordingly. Each nibble in that value denotes a port status. When you say you are touching only ax0, I see the correct nibble reflecting the change (bits[15:12]). From the values you showed here, looks like your SFP module is detected, but Link is still not. Is the link connected?

@manu, all the discussed changes (avoid repetitive logs, fix the debug sysctl, and your changes in P422) are done and updated the patch here.

1. Fixed the debug sysctl to be device instance specific
2. Minor Logging changes

Ok, weird that my modules doesn't work then no ?

I only have finisar ftlx8574d3bcl for now, do you have a known model that is supposed to work ?
Also can you make the printf not in a loop ?

Hi @manu, Thanks for trying the patch and making other necessary changes. Sorry, I missed to make that change MPASS (nrxqs == 2) when I submitted the v4 patch.

Does anyone have any more comments on this patch?

Changes :

1. Removed unwanted header files from unwanted places
2. Changed Rx path to use 2 Freelist instead of 1. Rx descriptors holds header and data buffers. Earlier use entries in one freelist were used to populate one descriptor. Now with 2 freelists, one entry from each freelist is used to populated on descriptor. This way the indexes will be in sync between driver and iflib

Hi @cem, Do you have any more comments on this patch?

Any more comments on this patch?

1. Removed "All rights reserved" and corrected years in the copyright
2. Added a minor change in i2c and phy-v2 files for link status update for multi-port

Change details :

1. Added SPDX-License Identifier for possible files with BSD-2-Clause-FreeBSD
2. Moved copyright tags out of the License text as per the comment
3. Changed to C style comments.
4. Removed commented code and added TODO tags.

Can we add an appropriate SPDX tag to the source files as well - e.g. /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ or similar?

Fine @cem. No issues.