libbe(3): rewrite init to support chroot usage

[kevans]

Summary

libbe(3) currently uses zfs_be_root and locates which of its children is currently mounted at "/". This is reasonable, but not correct in the case of a chroot, for two reasons:

• chroot root may be of a different zpool than zfs_be_root
• chroot root will not show up as mounted at "/"

Fix both of these by rewriting libbe_init to work from the rootfs down. zfs_path_to_zhandle on / will resolve to the dataset mounted at the new root, rather than the real root. From there, we can derive the BE root/pool and grab the bootfs off of the new pool. This does no harm in the average case, and opens up bectl to operating on different pools for scenarios where one may be, for instance, updating a pool that generally gets re-rooted into from a separate UFS root or zfs bootpool.

While here, I've also:

• Eliminated the check for /boot and / to be on the same partition. This leaves one open to a setup where /boot (and consequently, kernel/modules) are not included in the boot environment. This may very well be an intentional setup done by someone that knows what they're doing, we should not kill BE usage because of it.
• Eliminated the validation bits of BEs and snapshots that enforced 'mountpoint' to be "/" -- this broke when trying to operate on an imported pool with an altroot, but we need not be this picky.

Test Plan

• Run bectl... anything, and ensure that rootfs/bootfs/root get populated correctly
• Have @philip test on his bootpool and encrypted zpool chroot setup

[allanjude]

This is much better.

Not relying on the kenv should also solve issues that users booting UEFI were sometimes having.

[philip]

My setup is a remote headless machine booting from a zfs bootpool. Under normal circumstances, I decrypt the rootpool, kenv vfs.root.mountfrom=encrypted/ROOT/default and reboot -r. For upgrades, I'd like to decrypt the encrypted rootpool and chroot into its BE and run bectl in there.

With this patch, bectl *almost* works for me.

root@bootpool:~ # sh scripts/decrypt_disks.sh
root@bootpool:~ # zpool import -o altroot=/mnt encrypted
root@bootpool:~ # zfs mount encrypted/ROOT/default
root@bootpool:~ # mount -t devfs devfs /mnt/dev
root@bootpool:~ # chroot /mnt
root@bootpool:/ # bectl list
BE      Active Mountpoint Space Created
default NR     /mnt       1.36G 2018-11-16 17:31

So far so good -- but note that mountpoint is /mnt and not /. Proceeding, things blow up:

root@bootpool:/ # bectl create upgrade
mountpoint is not "/"
mountpoint is not "/"
mountpoint is not "/"
failed to create bootenv upgrade

I thought about being clever and jailing into the encrypted BE rather than chrooting:

root@bootpool:~ # jail -c path=/mnt mount.devfs name=upgrade host.hostname=upgrade ip4=inherit enforce_statfs=1 exec.poststart="zfs jail upgrade encrypted/ROOT" exec.start=true persist
root@bootpool:~ # jexec upgrade
root@upgrade:/ # bectl list
BE      Active Mountpoint Space Created
default NR     /          1.36G 2018-11-16 17:31
root@upgrade:/ # bectl create upgrade
mountpoint is not "/"
mountpoint is not "/"
mountpoint is not "/"
failed to create bootenv upgrade

That fails because libbe gets the mountpoint from ZFS rather than from the system. That's probably not too difficult to fix. :)

[kevans]

Ah, this is actually a thing Allan pointed out was bogus a couple days ago. I just need to axe ~20 lines- I think it fails in your case because with altroot, the mountpoint property really won't read as / but ZFS will do the right thing anyways.

[kevans]

Eliminate all of the bits that care about the mountpoint being "/" -- there were exactly two, in validating BE snapshots and in validating that a BE was valid.

[philip]

Woo. That almost works!

I can create, destroy, mount and jail BEs in the decrypted pool. When I try to umount though, it fails:

root@bootpool:/ # bectl umount upgrade
unknown error
failed to unmount bootenv upgrade

This is probably a chroot artefact more than anything else, because:

root@bootpool:/ # umount /mnt
umount: unmount of /mnt failed: Device busy
root@bootpool:/ # mount
[...]
encrypted/ROOT/upgrade on /mnt/mnt (zfs, local, noatime, nfsv4acls)
root@bootpool:/ # umount /mnt/mnt

Trying in a jail instead of a chroot:

root@bootpool:~ # jail -c path=/mnt mount.devfs name=upgrade host.hostname=upgrade ip4=inherit enforce_statfs=1 allow.mount=1 allow.mount.zfs=1 exec.poststart="zfs jail upgrade encrypted/ROOT" exec.start=true persist
root@bootpool:~ # jexec upgrade
root@upgrade:/ # bectl mount upgrade /mnt
successfully mounted upgrade at /mnt
root@upgrade:/ # bectl umount upgrade

Works fine!

I think this can be committed.

I did find an unrelated issue though. If I try to bectl jail inside the jail (because clearly I'm crazy), it fails:

root@upgrade:/ # bectl jail upgrade
unable to create jail.  error: 1

That's probably fine (but it would be cute it that worked). However, after it fails, it leaves the BE mounted on /tmp:

root@upgrade:/ # mount
encrypted/ROOT/upgrade on /tmp/be_mount.Mo6Z (zfs, local, noatime, nfsv4acls)

Probably a cleanup path that isn't quite complete.

[kevans]

Ok, good! The be_unmount can use some cleanup of its own, though, which might help a bit. Right now we use getmntinfo(2), locate the mountpoint for the requested BE, then unmount(2) it... this can all be simplified into zfs_open/zfs_unmount/zfs_close and probably be more robust from ZFS' perspective and might also fix your problem.