Page MenuHomeFreeBSD

gmultipath: make physpath distinct from the underlying providers'
ClosedPublic

Authored by asomers on Apr 22 2021, 10:38 PM.

Details

Summary

zfsd uses a device's physical path attribute to automatically replace a
missing ZFS disk when a blank disk is inserted into the same physical
slot. Currently gmultipath passes through its underlying providers'
physical path attribute. That may cause zfsd to replace a missing
gmultipath provider with a newly arrived, single-path disk. That would
be bad.

This commit fixes that problem by simply appending "/mp" to the
underlying providers' physical path, in a manner similar to what geli
already does.

Sponsored by: Axcient

Diff Detail

Repository
rG FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This revision was not accepted when it landed; it landed in state Needs Review.May 6 2021, 6:33 PM
This revision was automatically updated to reflect the committed changes.

I'm sorry for missing it earlier, but if there are actually several paths through the different HBAs and enclosures, what of the paths do you report? Random currently active one? Wouldn't it be better to not pass it through at all, unless we can somehow map all paths into something shared?

In D29941#676861, @mav wrote:

I'm sorry for missing it earlier, but if there are actually several paths through the different HBAs and enclosures, what of the paths do you report? Random currently active one? Wouldn't it be better to not pass it through at all, unless we can somehow map all paths into something shared?

It reports the currently active path's physical path, yes. That works with dual-path SAS backplanes, where both paths by design have the same physical path string. I don't know about Fibrechannel, iSCSI, or FCoE multipathing.

Excuse me if I am wrong, I don't have any multipath environment now to make sure, but I remember that SES on different expanders of dual-port backplane report different IDs.

In D29941#676901, @mav wrote:

Excuse me if I am wrong, I don't have any multipath environment now to make sure, but I remember that SES on different expanders of dual-port backplane report different IDs.

Hm. I thought that CAM used the Enclosure Logical Identifier to generate the physpath. It *should* do that. But now that I double-check; you're right. CAM is using the Addressed Logical Unit SAS ID instead. Maybe it was a Spectra-specific patch to use the ELI that never got upstreamed. @ken could you please check how SpectraBSD generates the physical path? You can use commands like

sudo diskinfo -p /dev/da0
sudo sg_ses -p1 /dev/ses0 | grep -i 'enclosure.*logical'
sudo sg_inq -p di /dev/ses0
In D29941#676901, @mav wrote:

Excuse me if I am wrong, I don't have any multipath environment now to make sure, but I remember that SES on different expanders of dual-port backplane report different IDs.

Hm. I thought that CAM used the Enclosure Logical Identifier to generate the physpath. It *should* do that. But now that I double-check; you're right. CAM is using the Addressed Logical Unit SAS ID instead. Maybe it was a Spectra-specific patch to use the ELI that never got upstreamed. @ken could you please check how SpectraBSD generates the physical path? You can use commands like

sudo diskinfo -p /dev/da0
sudo sg_ses -p1 /dev/ses0 | grep -i 'enclosure.*logical'
sudo sg_inq -p di /dev/ses0

{blackpearl:/usr/home/kenm:!:0} diskinfo -p /dev/da8
enc@5003048001e6057d/elmtype@array_device/slot@14
{blackpearl:/usr/home/kenm:!:0} sg_ses -p1 /dev/enc0 | grep -i 'enclosure.*logical'

enclosure logical identifier (hex): 5003048001e6057f

{blackpearl:/usr/home/kenm:!:0} sg_ses -p1 /dev/enc1 | grep -i 'enclosure.*logical'

enclosure logical identifier (hex): 5003048001e942ff

{blackpearl:/usr/home/kenm:!:0} sg_inq -p di /dev/enc0
VPD INQUIRY: Device Identification page

Designation descriptor number 1, descriptor length: 12
  transport: Serial Attached SCSI Protocol (SPL-4)
  designator_type: NAA,  code_set: Binary
  associated with the Target port
    NAA 5, IEEE Company_id: 0x3048
    Vendor Specific Identifier: 0x1e6057d
    [0x5003048001e6057d]
Designation descriptor number 2, descriptor length: 12
  designator_type: NAA,  code_set: Binary
  associated with the Addressed logical unit
    NAA 5, IEEE Company_id: 0x3048
    Vendor Specific Identifier: 0x1e6057d
    [0x5003048001e6057d]

{blackpearl:/usr/home/kenm:!:0} sg_inq -p di /dev/enc1
VPD INQUIRY: Device Identification page

Designation descriptor number 1, descriptor length: 12
  transport: Serial Attached SCSI Protocol (SPL-4)
  designator_type: NAA,  code_set: Binary
  associated with the Target port
    NAA 5, IEEE Company_id: 0x3048
    Vendor Specific Identifier: 0x1e942fd
    [0x5003048001e942fd]
Designation descriptor number 2, descriptor length: 12
  designator_type: NAA,  code_set: Binary
  associated with the Addressed logical unit
    NAA 5, IEEE Company_id: 0x3048
    Vendor Specific Identifier: 0x1e942fd
    [0x5003048001e942fd]

It is entirely possible that your memory is correct, and I messed it up in a subsequent merge.

If you could refresh my memory about where the physical path is getting set (I haven't been in there in quite a while), I can look back through the history and see if it has changed. I'm also happy to give you guys a copy of the current ses/enc driver. Merging it is a pain, and it would be great if we could upstream the changes.

Yep. That's wrong. In 14.0-CURRENT, the function that sets physpath is ses_set_physpath, in cam/scsi/scsi_enc_ses.c.

Yep. That's wrong. In 14.0-CURRENT, the function that sets physpath is ses_set_physpath, in cam/scsi/scsi_enc_ses.c.

Ok, it looks like it has been that way (using the device ID from page 0x83) since Justin wrote it in 2013.

I'm still happy to share the code we have, if you guys are going to do changes to SES, it would be nice to get some of our changes upstream. :)