Yes, this is a better way.

I'll admit very little familiarity with the testing framework. If there's a standard to show a test as skipped that doesn't indicate a problem, that sounds best. But if it just claims that it's passed, it would seem better to just not run it. Though both is probably a good idea: still have it pass (like if the test was built at another time), but don't build it on a jail-less system.

Here's the latest diff to address concerns so far (except those that request proper documentation).

In D43696#996617, @bz wrote:

Also, can we please have a man page?

Sure, looks helpful for just such a situation.

Looks good to me!

c) Or something else what I have not spotted yet :)

c) Jamie wasn't thinking and of course you don't need it for read-only.

You'll want to add CTLFLAG_PRISON to the sysctl flags.

In D43476#992839, @igor.ostapenko_pm.me wrote:

@jamie, does it look as an acceptable feature to introduce security.jail.children.max RO sysctl which reflects the current prison's p_childmax? If it does then I will help to implement it.
P.S. Probably you know existing "legal" ways to retrieve current prison's children.max w/o additional code to introduce.

I'm not sure why this limit exists in the first place (it predates me). I suppose it's just for neatness' sake, with the idea no one would have more jails than that anyway. But as long as it's around, it might as well be known.

That's fine if there's a use for it, such as to quiet errors.

In D42672#973897, @rmacklem wrote:

This version of the patch acquires a shared lock on
allprison_lock (which looks sufficient to ensure the
jail does not go from alive to dying.
It also acquires pr_mtx to check for prison_isalive(),
although I am not 100% sure it is needed?

In D42672#973775, @rmacklem wrote:

In D42672#973753, @markj wrote:

Since vfs_exjail_delete() is called when there are no processes running in the jail, a caller of vfs_exjail_clone() cannot be running in that jail.

It's not clear to me that the first part is true. In particular, prison_deref() first calls prison_deref_kill(), which calls prison_cleanup() and thus vfs_exjail_delete(); later it kills processes within the jail.

Hmm, unless I'm missing something, this is at odds with the comment in vfs_exjail_delete(). Perhaps we need to check the prison state when setting mnt_exjail? i.e., refuse to set it if the jail is dying.

Well, the comment on vfs_exjail_delete() states that no processes are in the
prison, but maybe the comment is bogus?
Anyhow, I think this version might be ok, but hopefully you or jamie@ can
confirm this?

My understanding (which could be wrong) is that a prison cannot go from
active to dying when there is a p_uref held on it. If that is the case, then
prison_proc_hold()/prison_proc_free() should ensure that it remains alive
until after the cloning is done, I hope?

The command parameters (including "command" itself) are well established as being run during jail setup and teardown. I would expect a lot of existing configuration to have problems with the command being run when the jail has already started up. While it makes sense on the command line, I don't want to break the connection between file configuration and command line configuration (more than it's already broken).

Nothing new, though I just updated the diff against the latest sources.

The important part of this clearly good.

Looks good to me. But then the original that did the direct cr_uid check looked apparently good to me too, so take it for what it's worth.

Commited in eb5bfdd06565. I forgot to add the review to the commit message :-/

I've committed the "jails can include jails" and "use the recursive parser" bits separately. This new diff is just the part that handles the includes.

In D40188#919081, @crest_freebsd_rlwinm.de wrote:

Just a small nitpick: I would prefer a macro #define MAX_INCLUDE_DEPTH 32

Simple include-loop prevention with via a maximum depth counter.

In D40262#917102, @nyan_myuji.xyz wrote:

However right now the kern.elfXX.fallback_elf_brand are tied to 2 global variables which their name are generated by the __elfN(xxx) macro. We could technically remove them and tie all 3 to the jail one? Maybe that belongs to a separate review?

Rather than add a separate value different from the kern.fallback_elf_brand sysctl, it would make sense for the jail parameter to be tied to the sysctl itself, such as the securelevel parameter is. It's complicated somewhat by the fact that there are three similar sysctls.

True, they're not handled. I took my include inspiration from newsyslog (which has includes that also support globbing), and there it's also just a simple matter or running whatever it's told to include. It's kind of a footgun situation, where it's generally good enough to trust the administrator not to make such a loop. I did it for depend loops, but only because that's kind of elemental in building an acyclic directed graph.

New and improved diff :-)

I just created D40188, which is my take on solving this issue. It goes the include-file direction, and doesn't require and changes to rc.conf, since jail.conf is where such changes are made. The jail(8) command line also remains untouched.

s/novnetjail/nojailvnet/

This makes perfect sense to me. The original version only set redo_ip[46] provisionally , and I missed that the patch changed that.

In D38144#871980, @rmacklem wrote:

Oh, I remember jamie@ mentioning the jail cleanup
method.

Would the call to the PR_METHOD_REMOVE function
be the right place to get rid of the credential references
in the mountlist?

I defined a function called vfs_exjail_delete(), which I currently
call from the nfsd'd OSD PR_MOETHOD_REMOVE which
releases credentials and set mnt_exjail NULL for all cases
matching the prison argument.
Maybe this function should be called from within kern_jail.c
via prison_cleanup()?

In D38371#872242, @mjg wrote:

I would expect killing the jail to *fail* if there any exports active. You can keep the counter of them in struct prison to avoid the need to scan anything.

With the assumption that nfsd in jail is *disabled* by default, this would not violate POLA.

The alive check raises a question: what if instead of a loop in vfs_export that checks prisons, how about a loop in prison_cleanup that checks exports? It could go through the mount list and any exports that belong to it can be refiled for the parent.

Sure, this one's no biggie.

I don't especially like the name pr_ident, which isn't readily differentiated from pr_id. Perhaps pr_permid? In a perfect world, I wouldn't have allowed prison IDs to be reused, but it's too late to stuff that horse back into the barn. Probably - I have considered actually changing the policy, but it would have POLA problems and would need a slow rollout with proper deprecation warnings.

In D38144#867425, @rmacklem wrote:

I noticed that there are old versions of "struct prison"
in sys/jail.h. Is this necessary when "struct prison" is
revised?

It took me longer to remember what I had done in the first place than it did to understand the new logic. I like this.

Yes, the warning is sufficient. As I mentioned, there are other cases where a jail is allowed to be created as long as its options are allowed, even if it turns out not to be "fit for purpose".

The thing about the vfs_opterror, is the error message only applies when the jail_set(2) fails (or at least it's only checked then). So If you want to still allow the jail to be created and only give a warning, then printf() is the way to go.