This passed beyond the point where I could add any meaningful feedback a while ago, but I wanted to just chime in briefly to thank everyone who worked on it. I'm really looking forward to playing with these images!

This split looks good to me. I'd also be happy if these were just pushed to public container registries, rather than to mirrors.

Thanks. It would also be nice to have an intermediate layer that is all of this except the toolchain, since the toolchain is very large but not needed for a lot of these use cases.

LGTM, assuming I understand the memory management correctly. I would be happier if something zeroed the gl_errblk field at the end of the call, because otherwise it is a dangling pointer. I don't *think* there's any code path that can reach it, but it's good practice to ensure that any pointer is either to a valid object or null.

This is also missing an update to the man page. The HISTORY bit should mention that glob_b first appeared in OS X 10.6.

I'm not quite sure what this is doing, but it isn't following the pattern that the rest of libc uses. We have DECLARE_BLOCK to declare a block type in a compiler-agnostic way and then CALL_BLOCK to call it. You don't need most of the #ifdef __BLOCKS__ bits at all, they're all handled by the abstraction layer.

This probably needs an exp run, I don’t know how many ports depend on this behaviour.

LGTM.

Address some more review comments.

Provide an option to disable caching.

Fix some review comments, remove a stale comment.

Thanks @asomers, I now understand FUSE slightly more, and that let me delete around 5% of the userspace code.

Address asomers' review.

Add a default BMAP. The kernel's FUSE seems to require it.

Fix more review comments.

min -> qmin

Address some review comments.

And, indeed, reverting this commit locally fixes it. Please can it be reverted and re-landed after testing with make packages?

A report on the pkg-base list suggests that this broke make packages. I am seeing the same breakage, which indeed does look like it's dying in the place modified by this commit:

In D45569#1039675, @brd wrote:

If 95% (or 99% or whatever) of our users don't need this, is it worth imposing this somewhat confusing question on the rest?

Why not change the text of the question to ‘are you dual booting Windows on this computer?’

After this change, ktrace output is littered with 'CAP system call not allowed: $SYSCALL' on systems w/o capsicum enabled

It looks as if this landed without a man page?

Oops!

I created this patch to make the Capsicumization experience less intimidating for inexperienced developers. Both David and Mariusz may not be the target audience for this change

It's doable in principle, but in practice dtrace's inability to resolve backtraces in the face of fork/exec makes it mostly unusable

Are these events exposed to DTrace? When sandboxing, the thing I really want is a stack trace in userspace at the point where the violation happened. If so, it would be great to include a script that logged them. Ideally with an option of an explicit start marker so you can put in a fake cap_enter and be told what you still need to fix.

To summarize the patch very briefly, this lets you ktrace an application that does not run in capability mode, and ktrace will log all events which would have triggered a Capsicum violation.

Capsicum-related syscall changes from 2014 on, for reference:

This looks like it fixes the capsicum bits. I believe we're still missing correct auditing events. In kern_{read,write}*, these are handled in dofile{read,write} but, since copy_file_range bypasses these calls, they need to be added explicitly.

Ensure that copy_file_range(2), like pread(2)/pwrite(2), requires the CAP_SEEK capability due to its offset argument.
This alignment prevents unauthorized file offset manipulations by processes.

One way out would be to atomic_swap the head to NULL.

Is there a reason to write new code that uses atomics and not use C11 atomics? We’ve had support for them for several releases now and it makes code much harder for new developers to understand if it uses nonstandard features for things that are part of the standard.

It looks as if this landed without a man page?

LGTM.

Thanks. You can enable fwcfg by -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu. You can also add additional item with the -f option. It has the same syntax like the -fwcfg option of qemu (https://www.qemu.org/docs/master/specs/fw_cfg.html#externally-provided-items).

Thanks. I've implemented a NetBSD-compatible kernel interface and tested it with the NetBSD FUSE filesystem:

Yes, exactly. It looks as if NetBSD has such a driver (https://man.netbsd.org/qemufwcfg.4), along with a FUSE FS the shared files into the filesystem. If anyone is working on such a thing for FreeBSD, it would be helpful, otherwise I will have a go next time I have some spare time.

Do you have plans to add the guest part of this device? Currently, Podman uses this mechanism to pass configuration information (specifically, accounts to create and ssh keys) to VMs that it creates to run containers. It would be possible to add an alternative mechanism for FreeBSD (though it’s not clear what that mechanism should be) but if there’s an existing device to grab this config and provide it to userspace that someone is working on then that would be much easier to use.

This looks fine to me. Assuming I'm reading the code correctly, this change is only for targets where int64_t is long, so there should be no change in any C++ / ObjC name mangling as a result of this. It's a bit curious that we explicitly define it as a fixed-width type for 32-bit targets but long for 64-bit prior to this change, did I do that?

12 is getting close to EOL, 14 is close to release, so it's too late for a back-port to 12 to matter to me.

I've rebased this, addressed reviewer comments (except the one that wants me to change some whitespace in a way that I don't fully understand), and tested it again with the Verona process-sandbox code.

• Address some reviewer comments.

LGTM. I thought we had those functions, but I've never seen any code in the wild using them, so if we don't then we should get rid of any reference to them.

The goal is to have tcmalloc natively working with full capacity and without further patching.

I would still like to see a design document and some design review before this is added. This feature in Linux has been quite controversial. I can see a case for copying a Linux feature that is widely used, or for adding a feature in the Linuxulator that is needed by certain workloads, but this doesn't seem to meet either of those requirements (not widely used, not being added to the Linux ABI layer here, only as a native syscall).

Fantastic, thanks!

This is because the controlling expression always undergoes lvalue
conversion first, dropping any cv-qualifiers

In D33248#764167, @kib wrote:

Don't you want to have core dumped for unhandled SIGCAP?

Ah, sorry, I thought these were additional changes. Looking more carefully, these all seem to be things that we've done upstream already.

Please can you raise a PR upstream that adds a macro to select between the two ABIs, so that we're not carrying a diff in the FreeBSD version?

Are there any other blockers to committing this? @markj, please can you approve if not?

It's still not clear what the purpose of this is. It's not added to the Linux ABI. If we're adding a new FreeBSD syscall, there should be some design review or at least motivating use cases. Linux' rseq is mostly useless (far less useful than a lightweight userspace interrupt delivery mechanism or a resume-from-context-switch handler) and is *only* vaguely useful on Linux in combination with the fact that Linux has a lightweight get-CPU system call implemented in the VDSO that is cheaper than a CPUID (which is a serialising instruction and generally costs more than the saving of most of the win from doing per-CPU instead of per-thread things).

• Fix comma splice.

In D33248#752692, @markj wrote:

FWIW, I tried the tests for your process_sandbox (with this patch applied of course) and one test consistently fails:

markj@biggie> ./test-onebitsem-child

• Address code review comments.

• Fix copy-and-paste fail.

Again, this is clearly outside the design space of rseq.

In D32505#733678, @kib wrote:

Lets limit the discussion to rseq(2) and not to some future hypothetical design needed for CheriBSD (which is not FreeBSD).

In D32505#733659, @kib wrote:

In D32505#733654, @theraven wrote:

• For allocator hardening, we'd like to ensure that signals delivered while executing in the allocator don't expose internal allocator state, so we'd like a mechanism somewhat closer to Windows structured exception handling that allows signals to be redirected based on a specific IP range.

Why should this be done in kernel? More, I believe that Windows does not do it in kernel either. Last time I looked (I admit it was very long time ago) they have a single upcall from kernel to userspace for all that stuff. It is usermode duty to interpret signal source + find the corresponding entry in the exceptions ranges table and do the unwind.

More, I think that there is a strong reason why kernel should not do that. You probably need to distinguish between sync and async signals, and further classify them based on si_code before even taking the look at the unwinding, so that only the events you are prepared for, like GC barriers or whatever you know about, started your specific actions. Kernel should not know about all that details.

@brooks, can you take a look at this from a CheriABI perspective? The current version uses the Linux convention of assuming that uint64_t is a sensible type for memory addresses. We probably can't do that in CheriABI because it would allow you to register a jump address that would make other code jump to your handler (or, conversely, it may prevent you from setting a range in some situations where there's a PCC change in between the rseq setup and the destination).

Thanks for doing this, it will make porting our work to FreeBSD easier. There are some missing man pages (both for the new pmap interfaces and for the syscall).

Abandoning: This is probably not the right approach and @jhb is working on something similar so I'll wait for his version.

@brooks, this should also be useful for combining CHERI coprocesses with Capsicum (coprocesses are currently created with vfork + coexecve).

• Propagate the field on fork.

• Add missing siginfo(3) documentation.
• Propagate the field on fork.

• Add missing siginfo(3) documentation.

Let's try option 2 and see if it works :-)

I've never used git send-mail, how does it interact with Phabricator? If I do a rebase / squash to produce a single commit, is that sufficient? You should be able to then arc patch and git push.

Thanks. Please can you commit if you're happy with it?