If this works I'm fine with landing it.

If this fixes installs, we should probably get it into 14.3. Can someone with more ZFS expertise than me comment on this?

In D40816#1155652, @allanjude wrote:

I am not sure how the import of OpenZFS would have changed any of this, but, for MBR we hide the 256kb /boot/zfsboot file in the slack space between the end of the ZFS uberblocks, and the start of the data area (4 MiB offset from the start of the partition).

The first 1kb, is installed in the slack at the start of the partition.

So basically:

1. mbr reads first 1kb of partition
2. first 512b of zfsboot knows where to find the rest of zfsboot
3. the rest of zfsboot is installed at an offset of 1024x512b where the first part of zfsboot knows where to find it.

1. mbr actually reads the first 512b of the partition

However, this change seems to not trust ZFS to do what you claim above. Rather than letting ZFS start at the start of the MBR slice, it is creating an explicit gap so that s1a doesn't start at the the start of a slice but leaves a 2048 sector (1MB) "hole" (between the start of the MBR slice, and the first BSD label partition ("s1a"). It then places the rest of the zfsboot payload at an offset (1024 sectors in) to the rest of that "hole" before the actual ZFS partition starts. So, clearly it seems like OpenZFS is not preserving the same layout for some reason and this change is trying to avoid interacting with ZFS entirely. If we really want ZFS + MBR to work this way, we should just change zfsboot to stop assuming the 1024 offset so we can write zfsboot contiguously to the start of the MBR slice. Then we could at least leave a smaller "hole" prior to "s1a". I'm also not sure how wise it is to leave unallocated space in the BSD label at the start of the slice. If you happen to fat finger a gpart command in the future, you could end up adding a new partition in the "hole" and end up overwriting the zfsboot contents. :(

Given all that, I wonder if we shouldn't just punt and say that ZFS + MBR isn't supported anymore. It hasn't worked in years and is pretty fragile.

In D40816#1164504, @jhb wrote:

In D40816#1155652, @allanjude wrote:

I am not sure how the import of OpenZFS would have changed any of this, but, for MBR we hide the 256kb /boot/zfsboot file in the slack space between the end of the ZFS uberblocks, and the start of the data area (4 MiB offset from the start of the partition).

The first 1kb, is installed in the slack at the start of the partition.

So basically:

1. mbr reads first 1kb of partition
2. first 512b of zfsboot knows where to find the rest of zfsboot
3. the rest of zfsboot is installed at an offset of 1024x512b where the first part of zfsboot knows where to find it.

1. mbr actually reads the first 512b of the partition

However, this change seems to not trust ZFS to do what you claim above. Rather than letting ZFS start at the start of the MBR slice, it is creating an explicit gap so that s1a doesn't start at the the start of a slice but leaves a 2048 sector (1MB) "hole" (between the start of the MBR slice, and the first BSD label partition ("s1a"). It then places the rest of the zfsboot payload at an offset (1024 sectors in) to the rest of that "hole" before the actual ZFS partition starts. So, clearly it seems like OpenZFS is not preserving the same layout for some reason and this change is trying to avoid interacting with ZFS entirely. If we really want ZFS + MBR to work this way, we should just change zfsboot to stop assuming the 1024 offset so we can write zfsboot contiguously to the start of the MBR slice. Then we could at least leave a smaller "hole" prior to "s1a". I'm also not sure how wise it is to leave unallocated space in the BSD label at the start of the slice. If you happen to fat finger a gpart command in the future, you could end up adding a new partition in the "hole" and end up overwriting the zfsboot contents. :(

Given all that, I wonder if we shouldn't just punt and say that ZFS + MBR isn't supported anymore. It hasn't worked in years and is pretty fragile.

I agree that 'not supported anymore' is the best answer.

Things are complicated by the fact that zfsboot is 256KB, and only the first 8KB of the ZFS partition is "reserved" by ZFS for the bootcode. So we end up having the first 512b of zfsboot in the MBR, and it seeks to a known offset to find the rest of it. It is better that that be outside of the ZFS partition, as the later reserved area is repurposed for things like raidz expansion. And without using the GEOM debug flags, you can't update the bootcode that is within the ZFS partition while running.

There are too many issues here to make it a "good idea"

Removal in review D51028

Azure wants to store its own metadata in an unallocated 1MB before the first partition: https://learn.microsoft.com/en-us/partner-center/marketplace-offers/azure-vm-certification-faq#vm-images-must-have-1-mb-of-free-space

This is a bad idea for azure and a bad idea for the installer, and the fact that there are multiple overlapping use cases for this hack makes it even worse for each of them.