Rebased to HEAD

In D10447#245058, @imp wrote:

This review is a little out of date and is much more ambitious than what I've just done in https://reviews.freebsd.org/D11820 . which also sets down this path, but not as completely. I fear that will make this a little harder, but maybe not by too much.

In D9680#239393, @imp wrote:

Why not 50MB instead of 512kb? That's stupidly small and precludes any and all future use of UEFI programs as well as boot block bloat. bz2 compressed, the size difference is trivial.

Rebased to head, addressed reviewer comments

Rebased to master

Remove extra bcache init

Also note that the increase dosfs size needs to go in first, or else it will break the build.

Address style and whitespace issues

As long as the max size is increased, it should work with my downstream changes.

Addressed style comments

In D10447#236550, @allanjude wrote:

If you can address the feedback that is here, and remove the whitespace changes, I think this is ready to land.

If I understand this correctly, this is a system based on symmetric-key message authentication codes which allows specific files to be assigned specific a specific MAC, which is then checked when those files are loaded.

Rebase to HEAD

Rebase to HEAD

Rebase to HEAD

Rebased to HEAD

Resolved issues with booting, tested with successful boot on QEMU. This should work now, but still needs to be tested on real hardware.

Use currdev environment variable, and libstand's basic file API.

Actually, there is a bug hiding out somewhere in this that causes ZFS volumes not to be detected.

Remove unnecessary changes to handles.c

Actually, efi_register_handle and efi_handle_remove_dev can be deleted from this patch. They were an artifact of when I tried to add hot-plugging support to the EFI loader (it doesn't work because bcache doesn't support it)

Removed getdesc function for ZFS (this was moved into the getdesc patch)

Integrate allanjude's pull request

Rebased after alanjude's patch adding explicit_bzero to boot loaders

Add changes by Allan Jude, adding support to boot2

Good change for my pending keybuf work

I like the interface in lib/libefi/libefi.c

In D9687#200246, @imp wrote:

All the string functions suffer from the same problem we have with the original string functions you replaced: they don't properly implement character conversions. Please see the efivar stuff I did in -current to see what you need to do there. And you should put the functions I wrote there into libstand. While these functions work well enough for ASCIIish things, they fail for anything more complex. If we're going to rework things, let's do it correctly and reuse what was done correctly for efivar (who, to be fair, stole and reworked the code from Marcel's earlier efi on ia64 work).

Add some more utility functions

Update on this patch: I'm going to attempt to break it up into some smaller changesets to make it more manageable.

In D9575#199104, @cem wrote:

This diff lacks context. Please use arc or diff with -U999999.

In D9575#199104, @cem wrote:

Moved MODINFOMD_KEYBUF definition to sys/linker.h

Added hook to clear GELI keys out of the intake buffer at mountroot.

In D9575#198206, @allanjude wrote:

In D9575#198186, @eric_metricspace.net wrote:

An observation: as presently implemented, this would bypass the password check for any GELI volume that was detached and later attached again. This is probably not desirable, as it could break security models.

One way to deal with this is to disable the keys after they are used to decrypt a volume. However, it warrants discussion.

I think we should explicit_bzero the keys/password as soon as we are done with them. GELI should already be doing this everything before the patch.

An observation: as presently implemented, this would bypass the password check for any GELI volume that was detached and later attached again. This is probably not desirable, as it could break security models.

Addressed style issues.

btw, there is also D8581 - it does work on another dimension however, by reusing disk and part API, and providing already familiar output for user.

With partition - filesystem mapping we can get the device handle without much issue - the handle stack has handles from FS down to partition and disk handle and the api to access the next handle from stack is there.

With zfs the situation is much more complicated as it is dual disk and file system and we need interface to access BE list and to switch the "current" filesystem on top of the "disk", and UEFI api itself does not help us there. It is possible to create specific protocol on top of zfs pool to implement an mechanism to get information about BE's and to switch, but it means that instead of reducing complexity, we are adding more.... note that the dataset list specifying BE's can be quite long, on my dev host I currently do have 73 entries.

Also, as I already wrote, for human operating the machine, it is really important to know exactly what I'm going to boot from.

tsoome, could you try out the new changes on your complex disk setup and let me know what the output looks like?

Fixed issue with directory listing file types.

It seems that I forgot about the file type issue with ls. I'll address that next, but please test the current patch to make sure it addresses the naming issue adequately.

Added functionality to print out information about the filesystems with lsdev.

In D7589#180667, @tsoome wrote:

ok, got it built, and initial test. And there is this problem...

OK lsdev

EFI0:    EFI(SIMPLE_FILE_SYSTEM)
EFI1:    EFI(SIMPLE_FILE_SYSTEM)
EFI2:    EFI(SIMPLE_FILE_SYSTEM)
EFI3:    EFI(SIMPLE_FILE_SYSTEM)
EFI4:    EFI(SIMPLE_FILE_SYSTEM)
EFI5:    EFI(SIMPLE_FILE_SYSTEM)
EFI6:    EFI(SIMPLE_FILE_SYSTEM)
EFI7:    EFI(SIMPLE_FILE_SYSTEM)

net devices:

net0:

OK

Yep, we can skip dosfs there, and for very simple reason - if the firmware is not able to read system partition, there is no boot anyhow.

OK, based on others' reproduction of tsoome's issue on -hackers, I have a credible diagnostic theory.

tsoome, can I get you to build and test with the extra_logging branch on my github repo: https://github.com/emc2/freebsd

I've posted a CFT to -hackers, -current, and -amd64.

In D7589#180054, @tsoome wrote:

In D7589#180052, @eric_metricspace.net wrote:

In D7589#180051, @tsoome wrote:

I am getting rejects on patch, could you rebase on current head?:)

tsoome@freebsd-2:/usr/eric % find . -name \*.rej
./lib/libstand/stand.h.rej
./sys/boot/efi/boot1/zfs_module.c.rej
./sys/boot/efi/boot1/ufs_module.c.rej
./sys/boot/efi/boot1/boot_module.h.rej
tsoome@freebsd-2:/usr/eric %

Strange. I had rebased just before I sent this up. I am rebasing from the github master, if that changes things at all. The last commit in the history is an hour ago.

My github repo for this is here: https://github.com/emc2/freebsd/tree/efize_new

Be aware: I'm using rebase -i to squash everything down to one commit at this point, so there's a lot of forced pushes going into efize_new

that may explain, as my fbsd source instances are all against svn ... so could it be the git has been missing some updates?

There's probably some lag going on somewhere. I see Ed Maste did push a change to the EFI loaders.

It might make more sense to build off of my repo in order to diagnose the issue your setup found.

Note, i may have been hit also by my own bug... see D8644

In D7589#180051, @tsoome wrote:

I am getting rejects on patch, could you rebase on current head?:)

tsoome@freebsd-2:/usr/eric % find . -name \*.rej
./lib/libstand/stand.h.rej
./sys/boot/efi/boot1/zfs_module.c.rej
./sys/boot/efi/boot1/ufs_module.c.rej
./sys/boot/efi/boot1/boot_module.h.rej
tsoome@freebsd-2:/usr/eric %

Strange. I had rebased just before I sent this up. I am rebasing from the github master, if that changes things at all. The last commit in the history is an hour ago.

My github repo for this is here: https://github.com/emc2/freebsd/tree/efize_new

Be aware: I'm using rebase -i to squash everything down to one commit at this point, so there's a lot of forced pushes going into efize_new

that may explain, as my fbsd source instances are all against svn ... so could it be the git has been missing some updates?

In D7589#180049, @tsoome wrote:

I did build from current state, the patch did apply clean, build faile due to -lstand issue noted before, thats simple to fix.

Unfortunately the boot is not going well, last message is:

Initializing modules: FS Backend-

the '-' is spinner it seems. The VM has 4 disks, illumos mirror zfs on first 2, third has fbsd, 4th (for this test) has EFI system partition, and zfs and ufs partitions, zfs partition has bootfs with /boot directory.

In D7589#178522, @eric_metricspace.net wrote:

In D7589#178521, @tsoome wrote:

I did build from current state, the patch did apply clean, build faile due to -lstand issue noted before, thats simple to fix.

Unfortunately the boot is not going well, last message is:

Initializing modules: FS Backend-

the '-' is spinner it seems. The VM has 4 disks, illumos mirror zfs on first 2, third has fbsd, 4th (for this test) has EFI system partition, and zfs and ufs partitions, zfs partition has bootfs with /boot directory.

Failing in FS Backend would indicate that something's going wrong with the filesystem attach phase. The FS Backend initialization scans all device handles and attaches the right filesystem drivers.

If have been testing on my own laptops with 2 disks, one with an EFI system partition and a ZFS zolume, one with a ZFS cache and log.

Therefore, the most likely culprit is UFS, though multiple ZFS volumes or the illumos mirror could be a problem.

What VM are you using? The easiest way to hunt this down would be to replicate your setup locally. The second easiest would be to get you a patch with extra logging.

vmware fusion. The illumos zfs mirror by itself should not be problem as loader zfs reader can read it both ways without problems (illumos loader can access fbsd pools and vice versa), also multiple pools should not be an issue as such - and definitely is scenario for tests. UFS may be issue, altho in this test setup, its just empty, I had that idea to create /boot on it as next step:) There is also another potential pain point - the memory allocations - as boot1 is now linked with libstand, could it be the malloc/free used all around are interfering somehow...

In D7589#180037, @eric_metricspace.net wrote:

Replaced uses of AllocatePool with malloc. Tested with a UFS partition, no problem.

tsoome, could you see if this affects anything? If not, then email me directly to coordinate diagnosis of the bug your setup found.

I am getting rejects on patch, could you rebase on current head?:)

tsoome@freebsd-2:/usr/eric % find . -name \*.rej
./lib/libstand/stand.h.rej
./sys/boot/efi/boot1/zfs_module.c.rej
./sys/boot/efi/boot1/ufs_module.c.rej
./sys/boot/efi/boot1/boot_module.h.rej
tsoome@freebsd-2:/usr/eric %

Replaced uses of AllocatePool with malloc. Tested with a UFS partition, no problem.

In D7589#178527, @tsoome wrote:

I did build from current state, the patch did apply clean, build faile due to -lstand issue noted before, thats simple to fix.

Unfortunately the boot is not going well, last message is:

Initializing modules: FS Backend-

the '-' is spinner it seems. The VM has 4 disks, illumos mirror zfs on first 2, third has fbsd, 4th (for this test) has EFI system partition, and zfs and ufs partitions, zfs partition has bootfs with /boot directory.

In D7589#178522, @eric_metricspace.net wrote:

In D7589#178521, @tsoome wrote:

I did build from current state, the patch did apply clean, build faile due to -lstand issue noted before, thats simple to fix.

Unfortunately the boot is not going well, last message is:

Initializing modules: FS Backend-

the '-' is spinner it seems. The VM has 4 disks, illumos mirror zfs on first 2, third has fbsd, 4th (for this test) has EFI system partition, and zfs and ufs partitions, zfs partition has bootfs with /boot directory.

Failing in FS Backend would indicate that something's going wrong with the filesystem attach phase. The FS Backend initialization scans all device handles and attaches the right filesystem drivers.

If have been testing on my own laptops with 2 disks, one with an EFI system partition and a ZFS zolume, one with a ZFS cache and log.

Therefore, the most likely culprit is UFS, though multiple ZFS volumes or the illumos mirror could be a problem.

What VM are you using? The easiest way to hunt this down would be to replicate your setup locally. The second easiest would be to get you a patch with extra logging.

vmware fusion. The illumos zfs mirror by itself should not be problem as loader zfs reader can read it both ways without problems (illumos loader can access fbsd pools and vice versa), also multiple pools should not be an issue as such - and definitely is scenario for tests. UFS may be issue, altho in this test setup, its just empty, I had that idea to create /boot on it as next step:) There is also another potential pain point - the memory allocations - as boot1 is now linked with libstand, could it be the malloc/free used all around are interfering somehow...

In D7589#178521, @tsoome wrote:

I did build from current state, the patch did apply clean, build faile due to -lstand issue noted before, thats simple to fix.

Unfortunately the boot is not going well, last message is:

Initializing modules: FS Backend-

the '-' is spinner it seems. The VM has 4 disks, illumos mirror zfs on first 2, third has fbsd, 4th (for this test) has EFI system partition, and zfs and ufs partitions, zfs partition has bootfs with /boot directory.

IFC, also properly restored preferred functionality

The preferred partition functionality is restored. The boot1 program now obtains the device handle from the loaded image handle, and checks for the presence of an EFI_SIMPLE_FILE_SYSTEM interface after attaching all drivers before scanning all partitions. This will have the effect of preferring the boot device on which the boot1 program resides.

To proceed, I need guidance on what exactly should be done to restore the preferred functionality.

In D7589#162680, @manu wrote: