Index: projects/pnfs-planb-server/usr.bin/pnfsdscopymr/pnfsdscopymr.1 =================================================================== --- projects/pnfs-planb-server/usr.bin/pnfsdscopymr/pnfsdscopymr.1 (revision 334651) +++ projects/pnfs-planb-server/usr.bin/pnfsdscopymr/pnfsdscopymr.1 (revision 334652) @@ -1,100 +1,99 @@ .\" Copyright (c) 2018 Rick Macklem -.\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd June 2, 2018 .Dt PNFSDSCOPYMR 1 .Os .Sh NAME .Nm pnfsdscopymr .Nd copy or move a data storage file for a MDS file to a different DS .Sh SYNOPSIS .Nm .Op Fl r Ar mounted-on-DS-dir .Op Fl m Ar source-mounted-on-DS-dir destination-mounted-on-DS-dir .Ar mdsfile .Sh DESCRIPTION The .Nm command copies a data storage file for an MDS file from one DS to another DS. It is normally used to recover data files onto a repaired DS, but can also be used to manually migrate a data storage file from one DS to a different one. By default, the command will copy the data storage file for .Dq mdsfile to one of the other DSs to create a mirror of it. This might be done if the file was created before mirroring was enabled on the pNFS service and now needs to be mirrored. .Pp The following options are available: .Bl -tag -width Ds .It Fl r Ar mounted-on-DS-dir This option indicates that the data storage file should be created on the DS that is mounted on the directory .Dq mounted-on-DS-dir . It will only do the copy if there is an entry in the pnfsd.dsfile extended attribute that has an IP address of 0.0.0.0. See .Xr pnfsdsfile 1 for how to do this. This is normally done for all regular files via .Xr find 1 in order to recover the data storage files onto a repaired DS. .It Fl m Ar source-mounted-on-DS-dir destination-mounted-on-DS-dir This option indicates that the data storage file is to be migrated from the source DS mounted on the diectory .Dq source-mounted-on-DS-dir to the DS mounted on the directory .Dq destination-mounted-on-DS-dir . In this case, the data storage file will be removed from the source DS when the copy is completed. .El If the copy/migration is already done, the command will simply exit(0), so that it can safely be used on all regular files in the exported directory tree on the MDS. .Pp This command must be run on the MDS and a typical usage would be as an argument for .Xr find 1 for all regular files. .sp For example, if the repaired DS is mounted on /data3 and files previously stored on the repaired DS have had the DS's IP address set to 0.0.0.0: .br # cd .br # find . -type f -exec pnfsdscopymr -r /data3 {} \\; .Sh SEE ALSO .Xr find 1 , .Xr pnfsdsfile 1 , .Xr pnfsdskill 1 , .Xr nfsv4 4 , .Xr pnfs 4 , .Xr nfsd 8 .Sh HISTORY The .Nm command appeared in FreeBSD12. Index: projects/pnfs-planb-server/usr.bin/pnfsdsfile/pnfsdsfile.1 =================================================================== --- projects/pnfs-planb-server/usr.bin/pnfsdsfile/pnfsdsfile.1 (revision 334651) +++ projects/pnfs-planb-server/usr.bin/pnfsdsfile/pnfsdsfile.1 (revision 334652) @@ -1,134 +1,133 @@ .\" Copyright (c) 2017 Rick Macklem -.\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd March 11, 2018 .Dt PNFSDSFILE 1 .Os .Sh NAME .Nm pnfsdsfile .Nd display a pNFS data storage file's location(s) and/or modify the .Dq pnfsd.dsfile extended attribute for them .Sh SYNOPSIS .Nm .Op Fl qz .Op Fl s Ar dshostname .Op Fl c Ar old-dshostname,new-dshostname .Op Fl r Ar dshostname .Ar mdsfile .Sh DESCRIPTION The .Nm command displays the data storage file's location(s) for a pNFS service and/or modifies the .Dq pnfsd.dsfile extended attribute on the .Ar mdsfile . A pNFS service maintains a data storage file for each regular file on the MetaData Server (MDS) on one or more of the Data Servers (DS). If mirroring is enabled, the data storage file will be on more that one of the DSs. Unless command options are specified, this command displays the location(s) of the data storage file for the MDS file .Ar mdsfile . It must be used on the MDS and the .Ar mdsfile must be a file on the exported local file system and not an NFSv4.1 mount. This information is stored in the .Dq pnfsd.dsfile extended attribute for this .Ar mdsfile . The command line options allow the information in the .Dq pnfsd.dsfile extended attribute to be changed. .Pp The following options are available: .Bl -tag -width Ds .It Fl q This option suppresses printing of the DS file's location(s). .It Fl z This option specifies that the file handle field of the pnfsd.dsfile extended attribute is to filled with all zero bits. This forces the pNFS MDS to do a Lookup RPC against the DS to acquire the file handle to update it. Normally this will only be necessary after the DS file has been recovered from a backup, causing the file handle to change. .It Fl s Ar dshostname This option can be used with .Fl z so that the zeroing out of the file handle is only done if the DS server is the one specified by this option. .It Fl c Ar old-dshostname,new-dshostname This option allows a sysadmin to replace the host IP# for the DS in the pnfsd.dsfile extended attribute. The old-hostname must resolve to the IP# already in the pnfsd.dsfile extended attribute or the replacement will not be done. If the old-dshostname matches, then the IP# is replaced by the first AF_INET or AF_INET6 address that .Xr getaddrinfo 3 returns for the new-dshostname. Changing a DS server's host IP# should be avoided, but this option will allow it to be changed, if the change is unavoidable. .It Fl r Ar dshostname This option sets the IP address of the extended attribute entry for the .Ar dshostname to 0.0.0.0 so that it will no longer be used. .Pp This is meant to be used when mirroring is enabled and the .Ar dshostname DS is disabled, so that it can be re-enabled once it is repaired. This needs to be done for all files in the exported MDS tree where the data may not be up-to-date on the repaired DS when it is re-enabled. After being re-enabled, the command .Xr pnfsdscopymr 1 with the .Dq -r option will be used to copy the the file's data to this repaired DS and then update the extended attribute to use it. .Pp A typical use of this will be within a .Xr find 1 for all regular files in the MDS's exported tree. .sp For example, if the disabled DS is nfsv4-data3: .br # cd .br # find . -type f -exec pnfsdsfile -q -r nfsv4-data3 {} \\; .El .Sh SEE ALSO .Xr find 1 , .Xr pnfsdscopymr 1 , .Xr pnfsdskill 1 , .Xr getaddrinfo 3 , .Xr nfsv4 4 , .Xr pnfs 4 , .Xr nfsd 8 .Sh HISTORY The .Nm command appeared in FreeBSD12. Index: projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 =================================================================== --- projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 (revision 334651) +++ projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 (revision 334652) @@ -1,65 +1,64 @@ .\" Copyright (c) 2018 Rick Macklem -.\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd March 9, 2018 .Dt PNFSDSKILL 1 .Os .Sh NAME .Nm pnfsdskill .Nd disables a pNFS data storage server (DS) .Sh SYNOPSIS .Nm .Ar mounted-on-DS-dir .Sh DESCRIPTION The .Nm command disables one DS when mirroring is enabled. If one mirrored DS is malfunctioning, a system administrator may use this command on the metadata server (MDS) to disable use of this mirror. This command must be used on the MDS and the .Ar mounted-on-DS-dir must be the exact pathname used when mounting the DS on the MDS. .Pp If this command fails with .Dq Device not configured (ENXIO), it probably means that the DS has already been disabled due to an error either detected by the MDS or reported to the MDS by a client. .Pp The pNFS service should continue to run normally so long as the number of operational DSs is at least as many as the level of mirroring. .El .Sh SEE ALSO .Xr pnfsdscopymr 1 , .Xr pnfsdsfile 1 , .Xr nfsv4 4 , .Xr pnfs 4 .Xr nfsd 8 .Sh HISTORY The .Nm command appeared in FreeBSD12. Index: projects/pnfs-planb-server/usr.sbin/nfsd/pnfs.4 =================================================================== --- projects/pnfs-planb-server/usr.sbin/nfsd/pnfs.4 (revision 334651) +++ projects/pnfs-planb-server/usr.sbin/nfsd/pnfs.4 (revision 334652) @@ -1,179 +1,178 @@ .\" Copyright (c) 2017 Rick Macklem -.\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd March 26, 2018 .Dt PNFS 4 .Os .Sh NAME .Nm pNFS .Nd NFS Version 4.1 Parallel NFS Protocol .Sh DESCRIPTION The NFSv4.1 client and server provides support for the .Tn pNFS specification; see .%T "Network File System (NFS) Version 4 Minor Version 1 Protocol RFC 5661" . A pNFS service separates Read/Write operations from all other NFSv4.1 operations, which are referred to as Metadata operations. The Read/Write operations are performed directly on the Data Server (DS) where the file's data resides, bypassing the NFS server. All other file operations are performed on the NFS server, which is referred to as a Metadata Server (MDS). NFS clients that do not support .Tn pNFS perform Read/Write operations on the MDS, which acts as a proxy for the appropriate DS(s). .Pp The NFSv4.1 protocol provides two pieces of information to pNFS aware clients that allow them to perform Read/Write operations directly on the DS. .Pp The first is DeviceInfo, which is static information defining the DS server. The critical piece of information in DeviceInfo for the layout types supported by FreeBSD is the IP address that is used to perform RPCs on the DS. It also indicates which version of NFS the DS supports, I/O size and other layout specific information. In the DeviceInfo, there is a DeviceID which, for the FreeBSD server is unique to the DS configuration and changes whenever the .Xr nfsd daemon is restarted or the server is rebooted. .Pp The second is the layout, which is per file and references the DeviceInfo to use via the DeviceID. It is for a byte range of a file and is either Read or Read/Write. For the FreeBSD server, a layout covers all bytes of a file. A layout may be recalled by the MDS using a LayoutRecall callback. When a client returns a layout via the LayoutReturn operation it can indicate that error(s) were encountered while doing I/O on the DS. .Pp The FreeBSD client and server supports two layout types. .Pp The File Layout is described in RFC5661 and uses the NFSv4.1 protocol to perform I/O on the DS. It does not support client aware DS mirroring and, as such, the FreeBSD server only provides File Layout support for non-mirrored configurations. .Pp The Flexible File Layout allows the use of the NFSv3, NFSv4.0 or NFSv4.1 protocol to perform I/O on the DS and does support client aware mirroring. As such, the FreeBSD server uses Flexible File Layout layouts for the mirrored DS configurations. The FreeBSD server supports the ``tightly coupled'' variant and all DSs use the NFSv4.1 protocol for I/O operations. Clients that support the Flexible File Layout will do writes and commits to all DS mirrors in the mirror set. .Pp A FreeBSD pNFS service consists of a single MDS server plus one or more DS servers, all of which are FreeBSD systems. For a non-mirrored configuration, the FreeBSD server will issue File Layout layouts by default. However that default can be set to the Flexible File Layout by setting the .Xr sysctl 1 sysctl ``vfs.nfsd.default_flexfile'' to one. Mirrored server configurations will only issue Flexible File Layouts. .Tn pNFS clients mount the MDS as they would a single NFS server. .Pp A FreeBSD .Tn pNFS client must be running the .Xr nfscbd 8 daemon and use the mount options ''nfsv4,minorversion=1,pnfs''. .Pp When files are created, the MDS creates a file tree identical to what a single NFS server creates, except that all the regular (VREG) files will be empty. As such, if you look at the exported tree on the MDS directly on the MDS server (not via an NFS mount), the files will all be of size zero. Each of these files will also have two extended attributes in the system attribute name space: .Bd -literal -offset indent pnfsd.dsfile - This extended attrbute stores the information that the MDS needs to find the data storage file on a DS for this file. pnfsd.dsattr - This extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file. .Ed .Pp For each regular (VREG) file, the MDS creates a data storage file on one (or on each mirror for the mirrored DS case) of the DSs which will store the file's data. The name of this file is the file handle of the file on the MDS in hexadecimal at time of file creation. The data storage file will have the same file ownership, mode and NFSv4 ACL (if ACLs are enabled for the file system) as the file on the MDS, so that permission checking can be done on the DS. This is referred to as ``tightly coupled'' for the Flexible File Layout. .Pp For .Tn pNFS aware clients, the service generates File Layout or Flexible File Layout layouts and associated DeviceInfo. For non-pNFS aware NFS clients, the pNFS service appears just like a normal NFS service. For the non-pNFS aware client, the MDS will perform I/O operations on the appropriate DS(s), acting as a proxy for the non-pNFS aware client. This is also true for NFSv3 and NFSv4.0 mounts, since these are always non-pNFS aware. .Pp See .Bd -literal -offset indent http://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt .Ed .sp for information on how to set up a FreeBSD pNFS service. .Sh SEE ALSO .Xr pnfsdscopymr 1 , .Xr pnfsdsfile 1 , .Xr pnfsdskill 1 , .Xr nfsv4 4 , .Xr exports 5 , .Xr fstab 5 , .Xr rc.conf 5 , .Xr nfscbd 8 , .Xr nfsd 8 , .Xr nfsuserd 8 .Sh BUGS Linux kernel versions prior to 4.12 only supports NFSv3 DSs in its client and will do all I/O through the MDS. For Linux 4.12 kernels, support for NFSv4.1 DSs was added, but I have seen Linux client crashes when testing this client. For Linux 4.17-rc2 kernels, I have not seen client crashes during testing, but it only supports the ``loosely coupled'' variant. To make it work correctly when mounting the FreeBSD server, you must either patch the Flexible File Layout client driver with a patch like: .Bd -literal -offset indent http://people.freebsd.org/~rmacklem/flexfile.patch .Ed .sp or set the sysctl ``vfs.nfsd.flexlinuxhack'' to one so that it works around the Linux client driver's limitations. .Pp Since the MDS cannot be mirrored, it is a single point of failure just as a non .Tn pNFS server is.