In D45780#1044154, @kp wrote:

There's probably a few other places we want to do the pull-up too, such as the ethernet rules and the IPv6 rules.
Places like pf_isforlocal() might need an assert.

Yeah, having such case unhandled, for years I guess, makes a feeling of some intention there. That's the source of my doubts. Thanks for sorting this out.

The first iteration happened here: https://reviews.freebsd.org/D45762.

I’ve been working on this PR:
[14.0 CURRENT] PF recognizes encrypted IPSec traffic as coming from WAN. | NAT with IPsec Phase 2 Networks
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273198

It was a good trial of the idea against the whole FreeBSD Test Suite with promising results. The discussion of the Kyua jail support patch (https://reviews.freebsd.org/D42350) ended up with opt-in approach. Thus, marking this as closed.

Jail related error messages are corrected after FreeBSD based code separation. The GitHub PR has been updated respectively: https://github.com/freebsd/kyua/pull/224.

I've updated the GitHub PR respectively: https://github.com/freebsd/kyua/pull/224.

Make mandoc -T lint and igor report nothing for kyuafile.5 and kyua.conf.5.

The respective documentation update is added. Two man pages are patched: kyua.conf.5 and kyuafile.5.

Updated summary of this Differential to align it with the current implementation.

Documentation update and extra tests of Kyua itself are still postponed until the code freeze.

This is a re-worked version of the patch after the recent discussion on freebsd-hackers@ list:

Align the patch with the main.

Align the patch with the main.

Please, find the PR here: https://github.com/freebsd/kyua/pull/224.

Resolve the recent comments.

@markj @ngie , the current state of the patch seems to cover most of the comments and agreements. I guess we could land it and think of further priorities like FreeBSD specifics separation, jail naming improvements (if it's a must today), probably tests of the kyua itself, man updates, etc. So, the subsequent changes could be separate reviews to ease the cognitive load.

Migrate from hard-coded JAIL_MAX to dynamic security.jail.children.max sysctl. The latter was recently added to the main: https://reviews.freebsd.org/D43565.

@jamie, thanks for the commit.

May I ask for the help? To land the commit itself.

I think we can abandon this proposal in favor of the dynamic security.jail.children.max sysctl. Landing this patch has a non-zero chance to break something, it looks better to keep the things as they are.

In D43565#994366, @kevans wrote:

Is the plan still to use JAIL_MAX in the test work, or to switch to this since tests can be executed started in non-prison0?

In D43565#994045, @jamie wrote:

c) Or something else what I have not spotted yet :)

c) Jamie wasn't thinking and of course you don't need it for read-only.

A little test improvement with additional comment about the magic numbers.

In D43565#993561, @jamie wrote:

You'll want to add CTLFLAG_PRISON to the sysctl flags.

In D43476#993381, @jamie wrote:

That would be acceptable, though I would prefer a more general solution. There are many useful bits of information about the current jail that don't need to be hidden from users, and few that are worth keeping hidden. I've been thinking of allowing jail_get(jid=0, ...) to see a sanitized version of the current jail (no path, jid, name, other things that only make sense in the parent context).

The purpose reasoning and initial discussion was in https://reviews.freebsd.org/D43476.

This update gets wait_any() back to the working state after the recent manipulations with the if-else branching there.

I've quickly checked different vectors of how children.max could be retrieved by an app for its current prison:

• Jail library. I see no way to do it, it requires jail name or jid, which are unknown for an app. And even if it's known, the current (parent) jail is "invisible". No surprise here -- all according to the jail design.
• Jail syscalls. As long as the libjail is merely a wrapper around the syscalls it leaves an app with the same dilemma.
• sysctl security.jail.param.*. I see that the CTLTYPE_INT leafs are not implemented, they are always 0. As the code comments state it's simply a "menu" of existing params.

In D43476#991829, @imp wrote:

Just to comment on the jail thing... I'd love it if we could get kyua running well in a jail... it's the best test we have for qemu bsd-user testing...

I hope this update covers most of the @ngie points raised.

In D42350#991331, @ngie wrote:

I held off on this review for a while, but I wanted to chime in about a few things:

Thank you all for your time and the review.

There is a follow-up idea.

Most of the agreed changes should be covered by this version of the patch.
Open topics left:

• jail naming
• required_klds

In D42350#988878, @glebius wrote:

One thing that I really miss in our jail/vnet testing suite is memory leak control. An automated snapshot of 'vmstat -m' and 'vmstat -z', run test in a jail, destroy the jail and compare with new vmstat snapshot. At first approach this may lead to lots of false positive, which aren't real leaks, but with fine filtering of which types/zones we really expect not to change that it going to work. That will bring a lot of value to the testing automation.

In D42350#985843, @markj wrote:

I'm sorry for the delay in replying. I'm not very familiar with kyua code and it'll take me some time to fully understand the patch.

Thanks for your time and consideration.

So my vote is to keep the existing behaviour the default, and let networking tests opt in to the new feature.

Thank you and @melifaro. This is exactly what I was looking for -- to get it concluded and confirmed by veterans what is the best way for the project.

Thanks for your attention. Agree, the details matter. It's expected to receive more opinions and real use case claims when the kyua jail support patch gets detailed consideration and discussion. And the change of existing test suite configuration (this patch) may go another direction after that.

Yes, we have to break the Demeter here for the sake of the bug fix. Some refactoring could take place here, but I guess it’s a separate project.

A concise version of the reasoning:

• The assertions have been added.
• This update also proposes to apply the same assertion for the main two anchors (existing code).

Fair enough. Thank you. It was an expected outcome, it confirms my doubts. And I personally do not have production needs or something else to continue work on this and discuss possible alternatives or workarounds like exec.foo.

Sure, it makes sense. Please, consider the second version of the patch.

This update provides no functional change:

• It aligns with existing Kyua architecture by moving some parts out from utils::process::jail to engine::execenv::jail. The philosophy of utils::* is to have no relation to the outside entities like test program or test case name.

This update only renames the temporary script used from cd_exec.sh to kyua_cd_exec.sh.

I've done the next step and now reached the level of the whole FreeBSD test suite. And my testings yielded the 3rd version of the kyua patch:

• it polishes test skip logic due to introduction of execenv cleanup phase
• and makes is_exclusive metadata be omitted if execenv="jail", which sounds practical comparing to if it would not do so

My thoughts regarding the FreeBSD test suite itself. For now I see two options. I will use /usr/src/tests/sys/netpfil/ipfw as a real example I tested the options against, assuming it has divert tests which are ready to be run in a jail and existing fwd test which is not ready to be run in a jail and in parallel with other tests, and we should keep it as is for now.

Yes, good point. Actually, the comment is not accurate, the issue is not only with jail name clashing, at least the routing table is in play as well.

Short summary of the changes:

• resolved conflict between ipfw and pf if both are used and pf wants to do divert(4)
• pf tests are split onto two sets: with ipfw enabled and disabled

In D42142#962263, @kp wrote:

That was on amd64, yes. I don't usually see platform-specific issues in pf though.
The version was this patch on top of:

commit 257405d707d77bc55b38e7c2bb83b8a9247a86ae (HEAD -> commit)
Author: Emmanuel Vadot <manu@FreeBSD.org>
Date:   Thu Oct 12 09:32:32 2023 +0200 
    xilinx: reset: Remove debug printfs

In D42142#962202, @kp wrote:

===> Execution context
...

In D42142#962199, @kp wrote:

Two of the new tests seem to fail on my test VM:

divert-to:in_div -> passed [16.923s]
divert-to:in_div_in -> passed [7.643s]
divert-to:in_div_in_fwd_out_div_out -> failed: Test case body returned a non-ok exit code, but this is not allowed [10.114s]
divert-to:out_div -> passed [15.990s]
divert-to:out_div_out -> failed: atf-check failed; see the output of the test for details [17.350s]

Added the first draft of the respective tests over the agreed fix.

In D42142#961485, @kp wrote:

I think the rulenum started out in ipfw, where it means 'continue evaluating rules from this rule number'. Hence the increment, which caused it to skip the divert-to rule.

Yes, ipfw is more flexible and allows to re-enter at the next rule number, e.g. this is how ipfw nat was done years ago, before the kernel nat. Prehistoric times...

@imp , considering your experience that GitHub fork's main can be left out-of-date (it's obvious project specifics, GitHub is not the origin for committers), if you see that comparing to project mainstream main branch would give better results in more cases, then we could do it. This commit can be used as an example: https://github.com/ihoro/freebsd-src/commit/0d273e225a3ea6e66d4f80e9e1f76c9ddca1454c.

@imp , I've created a PR to trigger the test. It's https://github.com/freebsd/freebsd-src/pull/751. The test passed. It can be simply closed without merge.

@imp , regarding your question whether it's possible not to hard-code main. Well, with every iteration my thoughts go deeper and complicate the things :) Nevertheless, I've dumped all my current research results and thoughts about this. Please, find it within the SVG attached (this is draw.io). Excuse me for such, probably, unusual format. I decided to leave the long read as is with the sketches, maybe it will be found helpful to explain the details for others and bring additional ideas.

If I can be of any assistance, I could do the following. I'm trying to guess here that probably it will ease the process for you.