Page MenuHomeFreeBSD

ixgbe: add MDIO bus support
Needs ReviewPublic

Authored by adrian on Sat, May 3, 2:06 AM.
Tags
None
Referenced Files
F117124043: D50128.id155235.diff
Wed, May 14, 9:28 AM
Unknown Object (File)
Mon, May 12, 2:16 PM
Unknown Object (File)
Sun, May 11, 9:32 AM
Unknown Object (File)
Sun, May 11, 9:14 AM
Unknown Object (File)
Sun, May 11, 8:33 AM
Unknown Object (File)
Sun, May 11, 4:04 AM
Unknown Object (File)
Sat, May 10, 11:28 AM
Unknown Object (File)
Thu, May 8, 2:28 PM
Subscribers

Details

Reviewers
kbowling
Group Reviewers
Restricted Owners Package(Owns No Changed Paths)
Summary

This works enough to let me see the marvell switch on the MDIO bus.

It uses clause 22, which ixgbe's existing MDIO code doesn't currently
support, so it's implemented in a new source file.

Since mdio(4) is now required, add it where appropriate to GENERIC kernels.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 64044
Build 60928: arc lint + arc unit

Event Timeline

Owners added a reviewer: Restricted Owners Package.Sat, May 3, 2:06 AM
adrian requested review of this revision.Sat, May 3, 2:06 AM
adrian added inline comments.
sys/dev/ixgbe/if_ix.c
279

oops i need to delete this too

1215

and this is here because i'm not quite sure yet what the preconditions should be to actually /attach/ an mdio bus to an ix interface. Let's figure THAT out first. :-)

sys/dev/ixgbe/ixgbe_mdio.c
5

i likely need to bump this to 2024, since the code i grabbed came from dpdk

sys/dev/ixgbe/if_ix.c
262

I would like these pushed down the file quite a ways, maybe after ixgbe_if_i2c_req, and add function prototypes above as necessary.

In general this looks fine aside from the cleanups you noted..

if_ix.c can be consider the FreeBSD "driver" where we can host our own changes and that seems the way DPDK structured it too with the functions in the PMD so no particular problem. But for posterity, in general Intel likes to stuff generic hardware dependent stuff in ixgbe_{chip}.c and access it through ixgbe_api.c which would kind of make sense here but we might also consider those files "upstream" (i.e. we can replay DPDK commits or out of tree driver commits more freely to them) so you are following DPDK as "upstream" in this case.

sys/dev/ixgbe/if_ix.c
262

honestly, i'm thinking of also making an if_ix_mdio.c file and just putting these in there.

(Especially if i'm going to use this as an excuse to play with clause 45 MDIO bus stuff and add those routines as i can then speak to the on-chip PHY register space and use it as a dev platform.)

sys/dev/ixgbe/if_ix.c
1215

Another angle.. what happens if this is exposed to a device with no mdio bus?

sys/dev/ixgbe/if_ix.c
262

That would be fine too, if_bypass.c is similar in spirit as a local hack (Intel doesn't like bypass adapters..)

In general this looks fine aside from the cleanups you noted..

if_ix.c can be consider the FreeBSD "driver" where we can host our own changes and that seems the way DPDK structured it too with the functions in the PMD so no particular problem. But for posterity, in general Intel likes to stuff generic hardware dependent stuff in ixgbe_{chip}.c and access it through ixgbe_api.c which would kind of make sense here but we might also consider those files "upstream" (i.e. we can replay DPDK commits or out of tree driver commits more freely to them) so you are following DPDK as "upstream" in this case.

As far as i can tell, DPDK kinda "broke" the c22 routines later on in their development. The chipsets docs for the x550 state the c22 support isn't available, but the x553 backplane docs say it is. I had to go back over their git history to find the initial import. They eventually replaced em with c45 only versions in the PHY code and left the c22 versions in rte_pmd_ixgbe.c. They don't even have an API for doing generic MDIO through ixgbe_api :(

sys/dev/ixgbe/if_ix.c
1215

Yup, that's why I need to figure this out.

The linux upstream driver always attaches an mdio bus, but for the backplane chipsets it only attaches it to the lowest present backplane pci dev as it's shared between all MACs. (The firmware / hardware semaphore guards against overlapping use though.0

migrate mdio stuff to if_ix_mdio.[ch], add/cleanup my copyright.

erj added inline comments.
sys/conf/files
2295

Is the "ixv inet" entry here required; ixv shouldn't need this file, right? I'm still not familiar with what needs to be added where in these kernel compile config files.

sys/dev/ixgbe/if_ix.c
1215

What happens when there's supposed to be an mdio device but it doesn't get attached here? I'm assuming the mdio driver will complain about something first, and then nothing will happen here, silently? It'd be good to find a way to see if a device is expected to have an mdio bus from within the ixgbe driver, but there doesn't seem to be a universal-ish mechanism for this like in ixl with its device/function capabilities admin queue commands.

kgalazka added inline comments.
sys/dev/ixgbe/ixgbe_mdio.c
94

I need to double check but I think IXGBE_MSCA_OLD_PROTOCOL can be used only with X552/X553 devices, and if I understand correctly MDIO interface is exposed for all types of HW. I wonder if it won't cause any unexpected side effects if used eg. with E610 (https://reviews.freebsd.org/D50067) which has completely new FW.

152

I'm not 100% sure how this will behave if another thread already acquired this semaphore. Also it guards single read/write and some operations require accessing several registers. I think you need to grab a iflib ctx lock in ixgbe_mdio_(read|wrige)reg

sys/dev/ixgbe/if_ix.c
1215

it /looks/ like the linux driver always attaches the mdio device, no matter what?

I'm looking at drivers/net/ethernet/ixgbe/ixgbe_phy.c : ixgbe_mii_bus_init(), which creates it for any NIC that this driver probes/attaches, and ixgbe_mii_bus_init() is called in ixgbe_probe().

So, is linux's ixgbe driver in this particular probe/attach path only supporting a subset of what ours does? Or is there something else I'm missing?

sys/dev/ixgbe/ixgbe_mdio.c
94

Would you mind checking? I'm comparing this to what the linux driver is doing too to try and make sure we're not deviating from what they're doing with the hardware.

Thanks!

152

I'm not 100% sure how this will behave if another thread already acquired this semaphore. Also it guards single read/write and some operations require accessing several registers. I think you need to grab a iflib ctx lock in ixgbe_mdio_(read|wrige)reg

I was wondering about that. Ok, lemme go find the iflib ctx lock bits and add them in here. Thanks!

sys/dev/ixgbe/ixgbe_mdio.c
152

Hm, the only iflib locks are /in/ iflib. How am I supposed to interact with this?

Should the mdio bus be exposed /by/ iflib first?

sys/dev/ixgbe/ixgbe_mdio.c
152

I would assume if you hold the fw semaphore you'd have exclusive access to the adapter.. is the worry that callers might also have requested the semaphore before calling this?

In e1000 there are some gymnastics around ASSERT_CTX_LOCK_HELD and you can see where I prod it in a similar situation for accessing the NVM with struct sx *iflib_ctx_lock = iflib_ctx_lock_get(sc->ctx).. I forget the exact details why we had to do that, it might have been related to the watchdog (which used to trip a lot) triggering stalls, but it does serialize access to the hardware semaphore.

add iflib ctx lock, suggested by kbowling / erj

adrian marked 3 inline comments as done.

update copyright

adrian marked 2 inline comments as done and an inline comment as not done.Sat, May 10, 4:44 AM

ok, i've cleaned up a bit! whats left before it can be landed?

I would be somewhat curious why we need to serialize like this. Note that nothing else prodding the swfw in ixgbe is explicitly locked or asserted.