kyua: Add "prepare" command

[igoro]

This is the very first working version of the Project B: https://lists.freebsd.org/archives/freebsd-testing/2024-November/000395.html.

rr stands for:

• requirement resolver
• resolve requirements
• requirement resolving

Currently the interface is as follows:

• A new command kyua rr lists all available requirement resolvers
• kyua rr resolver1 resolver2 ... runs the resolvers listed
• kyua rr all runs all available resolvers
• kyua rr accepts the same --kyuafile and --build-root options as kyua test does
• A new option kyua test --rr=resolvers is added as a handy shortcut to resolve requirements and run tests, e.g. kyua test --rr kmods
• Whitespace-separated list of required modules can be declared as required_kmods on Kyua side and as require.kmods on ATF side (this is what is expected to be extended by the Project A to add skipping mechanism)
• A new resolver kmods is added exclusively for FreeBSD, it consumes required_kmods/require.kmods

At least the following things are in my todo:

• Add/update man pages
• Add new tests
• Analyze cases flagged by olivier@ where specific tests do kldunload
• Final commit log message
• GitHub freebsd/kyua fork PR

[markj]

As usual, I will quibble a bit about naming - the kyua subcommands are mostly english words, so "rr" is a bit odd. Why not "resolve" or even "resolve-requirements" (or "resolve-reqs")?

Another comment is about candidate requirements: this mechanism is useful for loading kmods, but I'm not sure how it'll work with other requirements. What does it mean to resolve require.memory? The only other resolvable requirement I can think of (and which you mentioned in the ML post) is packages, but tests just enumerate required programs, which doesn't tell you how to install them. For example, required.programs = nping requires one to install the nmap package, but we can't know that without additional metadata. Plus, the test environment might not have access to pkg repos.

Having this mechanism to load kernel modules is indeed useful, but I have a hard time seeing how it can be generalized. Please don't take that as an objection. This feature doesn't require a lot of code, so isn't too intrusive.

[igoro]

In D48087#1096189, @markj wrote:

As usual, I will quibble a bit about naming - the kyua subcommands are mostly english words, so "rr" is a bit odd. Why not "resolve" or even "resolve-requirements" (or "resolve-reqs")?

Thanks for that, we do need to stress the design here. Initially, I was thinking about something like "kyua resolve-required", where "required" is like an established word due to required_* metadata. That's fine for unattended cases like CI scripts or bricoler, but I was thinking it might end up being tedious to type, and shell completions might not always be available. Hence, I started the discussion with less obvious option to cover my doubts. What if we have a long name and a short alias as it frequently happens with CLI? Like "checkout" -> "co" and so on. As long as the subcommand expects a list of resolvers, we could consider another naming variant as well:

> kyua run-resolvers kmods pkgs
> kyua rr kmods
> kyua test --run-resolvers pkgs
> kyua test --rr kmods

Another comment is about candidate requirements: this mechanism is useful for loading kmods, but I'm not sure how it'll work with other requirements. What does it mean to resolve require.memory? The only other resolvable requirement I can think of (and which you mentioned in the ML post) is packages, but tests just enumerate required programs, which doesn't tell you how to install them. For example, required.programs = nping requires one to install the nmap package, but we can't know that without additional metadata. Plus, the test environment might not have access to pkg repos.

I was thinking about a new "required_pkgs" metadata. Also OS agnostic as "required_kmods" is. Just a whitespace-separated list of names. And the FreeBSD resolver named "pkgs" could read it and do the usual pkg install. Chasing packages via required_programs does not seem to be deterministic. If we provide an optional mechanism a specific test host may still use custom third-party software/builds and avoid "pkgs" resolver usage, e.g. some pre-cooked VM or OCI image etc. Thus, required_programs and required_pkgs could stay as separate mechanisms which could be used together if needed. Probably, "required_pkgs" should not provide any check-and-skip mechanism and should stay as a pure declaration of needs -- some tests may use both "required_pkgs" and "required_programs".

Having this mechanism to load kernel modules is indeed useful, but I have a hard time seeing how it can be generalized. Please don't take that as an objection. This feature doesn't require a lot of code, so isn't too intrusive.

Yes, good news is that it does not require a lot of code, and it's great to follow the Kyua architecture which tries to provide extensible sub-systems.

One of the key points I had in mind is that it should not run all resolvers by default. For instance, I have a prepared system and the only thing I need is "kyua test --rr kmods testprogram" due to modules are not there after reboot, and I want to avoid "pkgs" resolver spending my precious seconds to check that I have everything installed (or it may disrupt my setup with updates, or the test host has no Internet connection, etc). Such cases also tend to have some generic approach here. And the special "all" resolver runs all of them if we do need it.

There is no intention to add resolvers w/o actual demand, but I had the following resolvers in my mind, without giving it thorough thought yet:

• kmods
• pkgs
• pip? # e.g. to have scapy installed
• sysctls # some tests may want to have specific settings like turn-on-a-feature, e.g. sysctl kern.ipc.tls.enable=1 and sysctl kern.crypto.allow_soft=1
• thinking out loud: in the future we may want to prefetch some OCI images, similar to pkg install idea

[igoro]

By the way, now I'm leaning towards resolve subcommand as you mentioned:

> kyua resolve kmods
> kyua test { --resolve | -r } kmods testprog

Let's gather more opinions/votes before the next version of the patch.

[kp]

I generally share Mark's views. I was tempted to suggest we should rename it to 'load-kmods' or similar, but there's a somewhat reasonable path to it also being able to install required packages (even if that will require test modifications), so keeping the generic name seems sensible. If we do get to that point we should consider adding a 'kyua prepare' or something to take care of all of the requirements (i.e. kmods, packages) it knows about. That should let us run (most) tests with kyua prepare && kyua test on a new install. (That's very much a maybe someday sort of thing, we do not need that to proceed here.)

One trivial remark about 'pip? # e.g. to have scapy installed'. We have a scapy package and that's how I've always installed it. Right now it's py311-scapy-2.6.1.

[igoro]

Rename "kyua rr" to "kyua prepare"; Drop "kyua test --rr"

[igoro]

My current favorite is the prepare term proposed by Kristof. So, I've changed it in the latest version of the patch for the sake of further testing and discussion. Everything is open for renaming. The internals still use requirement resolver terminology, I could not quickly find the replacement and probably it's not needed due to "prepare" is kind of higher level, anyway this is the encapsulated part with the only exposure via CLI help output.

Also, I've dropped any change to kyua test. I think it's too soon to provide shortcuts there due to non-established yet general concept, or probably it's better to postpone it until actual demand. It might be better to keep prepare and test as separate things as I envisioned from the very beginning.

I've checked the points raised by Olivier (https://lists.freebsd.org/archives/freebsd-testing/2024-November/000396.html). It seems we change nothing here, it's like instead of listing the required modules manually as the maintainers of the CI configuration do today we will get the list from the declarations by the tests themselves. And those specific ipsec and opencrypto tests will do the usual kldload/kldunload things as today. The sys/netipsec/tunnel tests are already marked as is_exclusive, I guess we could do the same for the sys/opencrypto tests -- so that we make sure they do not play "kld[un]load" cards while there are others running. Of course, it means that these ipsec/opencrypto tests will not use required_kmods for modules they handle on their own. It seems that my current vision correlates with Kristof's "perhaps we don’t need to worry about that just yet" (https://lists.freebsd.org/archives/freebsd-testing/2024-November/000397.html).

A quick recap of the UI provided by the latest patch:

> kyua prepare
all                     Run all requirement resolvers
kmods                   Resolve required_kmods for FreeBSD

> grep -r kmods *
Kyuafile:atf_test_program{name="divert-to", execenv="jail", execenv_jail_params="vnet allow.raw_sockets", required_kmods="pf ipdivert iflib"}
Kyuafile:atf_test_program{name="mbuf", execenv="jail", execenv_jail_params="vnet allow.raw_sockets", required_kmods="ether pflog if_bridge"}
mbuf:   atf_set require.kmods if_epair if_bridge pf

> kyua prepare --dry-run all
kldload ether if_bridge if_epair iflib ipdivert pf pflog

> kyua prepare kmods
kldload ether if_bridge if_epair iflib ipdivert pf pflog

The above demonstrates:

• Kernel module requirement declaration on Kyuafile and ATF test case levels
• Built-in modules support (the iflib and ether are used just for the demo)
• The all resolver in addition to the kmods
• The --dry-run | -n option of kyua prepare command

[igoro]

Now the wording is away from the internals and closer to the end users:

> ./kyua help | grep prepare
  prepare                 Prepare env and resolve requirements before testing.

> ./kyua prepare
all                     Run all preparations
kmods                   FreeBSD: kldload modules declared using required_kmods metadata