kyua: Add execution plan concept
Needs ReviewPublic
Actions

Authored by igoro on Sun, Aug 30, 9:23 PM.

Details

Reviewers: None

Summary

The idea is that kyua now uses wait6() to collect wrusage per test and save it into a file, after full run of kyua test command. Next time, having such test profile on the disk (usually beneath ~/.kyua/profiles/) it can schedule testing in a special way based on the facts from the profile. To apply such special scheduling Kyua must be run with a newly added configuration parameter (-v execplan=time).

Obviously, it does not provide magic multiple times speedup, it seems to potentially provide -10-20% of runtime or so. But the idea is that running many iterations is expected to bring a cumulative improvement in the total runtime.

The first run is the usual one to generate a profile:
/usr/tests> time -h kyua -v parallelism=N -v execplan=time test

From the 2nd run , having the profile, it should schedule the tests differently:
/usr/tests> time -h kyua -v parallelism=N -v execplan=time test

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 76336
Build 73219: arc lint + arc unit

Event Timeline

igoro created this revision.Sun, Aug 30, 9:23 PM

Herald added subscribers: ziaee, imp. · View Herald TranscriptSun, Aug 30, 9:23 PM

igoro requested review of this revision.Sun, Aug 30, 9:23 PM

Harbormaster completed remote builds in B76336: Diff 185447.Sun, Aug 30, 9:23 PM

That's an interesting idea.
Running the pf tests (at parallelism=4) that knocks the runtime down from 16 minutes to 10 minutes, which is pretty impressive.

That said, I find myself mostly running tests in bricoler these days, and it won't help there because that's a new single-use VM every time, without the profile information. (Though perhaps we could teach bricoler to import existing profile information from the host system?)

igoro edited the summary of this revision. (Show Details)Mon, Aug 31, 12:37 PM

In D59276#1360076, @kp wrote:

That's an interesting idea.

Yeah, I think this is a good idea. My only concern is that it might eliminate some randomness from the test scheduling that helps probabilistically catch problems. I have not looked at all at the patch yet, nor at the existing scheduler: how does the patch affect determinism in the test scheduler?

Running the pf tests (at parallelism=4) that knocks the runtime down from 16 minutes to 10 minutes, which is pretty impressive.

That said, I find myself mostly running tests in bricoler these days, and it won't help there because that's a new single-use VM every time, without the profile information. (Though perhaps we could teach bricoler to import existing profile information from the host system?)

Do you ever use the freebsd-regression-test-suite-ci target? It does a few things on top of freebsd-regression-test-suite:

it keeps track of results across successive runs, so you'll have persistent results for run #1, #2, ...
after the test suite finishes, it'll rerun failed and broken tests in a new VM, without parallelism enabled, so that we can better detect flakiness
it can send email summaries after the test run finishes

It'd be easy to change that task to save the profile as well, and when preparing to start a test run, it could use the previously saved profile to seed the next run.

Thinking a bit more, we could even integrate it into plain freebsd-regression-test-suite: just save the profile after a test run, like we already save the results db, and use it when starting a new run if available. So, I think this won't be a problem. I'll try to integrate it in the next day or so.

In D59276#1360539, @markj wrote:

In D59276#1360076, @kp wrote:

That said, I find myself mostly running tests in bricoler these days, and it won't help there because that's a new single-use VM every time, without the profile information. (Though perhaps we could teach bricoler to import existing profile information from the host system?)

Do you ever use the freebsd-regression-test-suite-ci target? It does a few things on top of freebsd-regression-test-suite:

it keeps track of results across successive runs, so you'll have persistent results for run #1, #2, ...

after the test suite finishes, it'll rerun failed and broken tests in a new VM, without parallelism enabled, so that we can better detect flakiness

it can send email summaries after the test run finishes

I've never used that, no. I don't really have a need for that, although I suppose the automatic re-running of failed tests is interesting.

My own use case for bricoler is mostly for a clean test of a patch (so I'm confident I'm testing what I think I'm testing) and a really easy way of getting the KASAN/KMSAN configs tested.