Page MenuHomeFreeBSD

cam: save parent_dev in xpt_bus_register
ClosedPublic

Authored by imp on Jun 21 2021, 6:00 PM.

Details

Summary

Sponsored by: Netflix

Test Plan

Backstory: when I suggested to kibab we needed a cam_sim_alloc_dev() routine, I didn't realize it was already part of
xpt_bus_register(). It was unused there, though there was a parent_dev in cam_ed. This catches up and sets that.
A future commit will eliminate cam_sim_alloc_dev() and pickup the dev from the cam_ed device.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

imp requested review of this revision.Jun 21 2021, 6:01 PM
imp added reviewers: ken, mav, scottl.

While duplication of device_t pointers in bus and sim is weird, I still don't see a strong concept here. Do we have any SIM consisting of several device(9) devices? I don't know any. But considering it is already passed by many drivers we may keep it. Does the following bus argument means bus number within that device, or the number of the bus within the SIM? Right now it looks completely broken/useless to me, so we may invent some meaning for it on the way.

In case of single device(9) having several buses the bus number could reflect that fact, but then it has to be moved from sim to bus. Otherwise I am not getting its meaning on top of sim->unit_number and sim->path_id.

In D30846#694541, @mav wrote:

While duplication of device_t pointers in bus and sim is weird, I still don't see a strong concept here. Do we have any SIM consisting of several device(9) devices? I don't know any. But considering it is already passed by many drivers we may keep it. Does the following bus argument means bus number within that device, or the number of the bus within the SIM? Right now it looks completely broken/useless to me, so we may invent some meaning for it on the way.

Scott added the device_t argument back in FreeBSD 7 time frame when it was contemplated to move the cam devices to newbus devices. That hasn't happened, but all the driver were converted to pass this in.
The notion I've always used here, and seen in large part in the tree, was that this was the device_t, if any, that corresponded to the sim.
The bus number pre-dated Scott's change. It's documented in the SCSI architecture handbook we have as usually being 0, except when the SIM has multiple buses that are exposed with separate target/lun address spaces. mpt is one example for bus number, btw.

ctl's front end, iscsi both legitimately pass in NULL today. hptnr and hptrr likely could pass in a dev, but don't.

In D30846#694546, @mav wrote:

In case of single device(9) having several buses the bus number could reflect that fact, but then it has to be moved from sim to bus. Otherwise I am not getting its meaning on top of sim->unit_number and sim->path_id.

The different buses are currently used inside of xpt for reporting which scbus is connected to that bus as well as for some bus-specific traversal....

But that's where I get confused because it sets sim->bus_id to this value. And mpt's use of it is for two different busses (one for the volume bus, which may have disks, and one for the physical disk bus). And there mpt creates two different SIMs. mrsas does the same.

In D30846#694549, @imp wrote:

The bus number pre-dated Scott's change. It's documented in the SCSI architecture handbook we have as usually being 0, except when the SIM has multiple buses that are exposed with separate target/lun address spaces. mpt is one example for bus number, btw.

For the case of multiple buses per SIM, as I have told, it is stored in wrong place and can't work. It makes some sense through for the case of multiple SIMs per device_t (as in mpt(4)), except then it is even more weird to see parent_dev in bus with bus_id in sim.

In D30846#694551, @mav wrote:

For the case of multiple buses per SIM, as I have told, it is stored in wrong place and can't work. It makes some sense through for the case of multiple SIMs per device_t (as in mpt(4)), except then it is even more weird to see parent_dev in bus with bus_id in sim.

The two uses I could find in the tree were mpt and mrsas. Both of these create two SIMs. Both pass the same device_t, but different bus numbers. This is obscured in mpt by a macro which seems to be for support for pre FreeBSD 7...

ahc(4) creates two buses for the same device_t with the same lock, devq and methods. I'd think it could be a perfect case of multiple buses per SIM. But I don't know why it is done the way it it is done, except that sim->bus_id would not work right otherwise.

If the ahc(4) is not the perfect case for two buses per SIM, then I don't know why do we even have that differentiation.

Thinking for another second, since xpt_bus_register() assigns sim->path_id, then I can't even guess what is the use case for multiple buses sharing the target ID range. It looks like another thing to cleanup or formalize. "sim->path_id = new_bus->path_id = xptpathid()" in xpt_bus_register() looks weird.

In D30846#694555, @mav wrote:

Thinking for another second, since xpt_bus_register() assigns sim->path_id, then I can't even guess what is the use case for multiple buses sharing the target ID range. It looks like another thing to cleanup or formalize. "sim->path_id = new_bus->path_id = xptpathid()" in xpt_bus_register() looks weird.

ahc also allocates two sims. It uses bus 0 and 1 for the bus numbers, but only registers one bus per sim. This bus number is different than the path_id, which is allocated in the snippet you quoted.

This also tells me that a cam_find_sim interface I'm working on is incomplete because there can be more than one if I just use sim name an unit...

The model is that a sim represents the queueing and resource pool for one or more buses. If multiple buses have their own private resource and scheduling pools, then those buses need to have separate sims. If they share resources, then they need to share the same sim. AHC was weird because even though it had a split "TWIN" mode, apparently the split buses still had independent resource pools and queueing limits. The same was not true with cards that presented logical and physical drives like mpt and ciss; they still draw from the same resource pool, so they needed a single SIM and also need distinctive buses so the physical and logical target numbers don't conflict. This is really a detail of the driver and hardware, and is something that the driver writer should know and understand early in the design phase of development. Even though SAS controllers have multiple "ports", the queueing is still typically common to all ports on the device, so only a single bus is created on the sim.

This revision is now accepted and ready to land.Jun 24 2021, 2:02 AM

Thanks for the model... I may have to use parts of it in the CAM architecture update in the handbook if you don't mind..

This revision was automatically updated to reflect the committed changes.