tests/unix_seqpacket: provide random data pumping test with MSG_EOR

[glebius]

Summary

Allocate a big chunk of randomly initialized memory. Send it to the peer
in random sized chunks, throwing MSG_EOR at randomly initialized offsets.
Receive into random sized chunks setting MSG_WAITALL randomly. Check that
MSG_EORs where they should be, check that MSG_WAITALL is abode, but
overriden by MSG_EOR. And finally memcmp() what we receive.

[glebius]

The test will fail on CURRENT. It will be committed together with new unix/stream & unix/seqpacket implementation.

[asomers]

This looks good. But I have a few thoughts:

This test case is comprehensive, but it's also pretty complicated. Best practice is to start with a few minimal unit tests, each as small as can be, before adding stress tests like this one.

This test will execute a different sequence of operations every time. That will make failures hard to reproduce. If you're going to use randomness in a test case, you should do it deterministically. Could you please replace arc4random() with random() so you can use initstate()? I suggest generating a random seed value to start, printing that seed, and then supplying the seed to initstate.

Also, don't forget to pthread_join, and to free all of the allocations.

[glebius]

I'll make resources reclaim patch, thanks!

But I don't agree with going for deterministic random. That will reduce test coverage. For such kind of tests to get coverage tending to 100% you need either go with deterministic random and volume of data to pump needs to tend to infinity. Or you can go true random and then make number of runs tend to infinity. The CI itself gives us the latter for free.

Of course failures reported by CI (if any) won't be reproducible immediately. But they will be a red flag. Once I got a failure, I will run the test in a loop until reproduction.

[asomers]

In D43775#998578, @glebius wrote:

I'll make resources reclaim patch, thanks!

But I don't agree with going for deterministic random. That will reduce test coverage. For such kind of tests to get coverage tending to 100% you need either go with deterministic random and volume of data to pump needs to tend to infinity. Or you can go true random and then make number of runs tend to infinity. The CI itself gives us the latter for free.

Of course failures reported by CI (if any) won't be reproducible immediately. But they will be a red flag. Once I got a failure, I will run the test in a loop until reproduction.

Oh, I'm not suggesting that you run the test with a fixed seed every time. I'm suggesting that you run it with a different random seed every time, but print the seed. That way, a user can quickly reproduce the failure by modifying the test to use the fixed seed instead. Like this:

uint32_t state[32];
uint32_t seed = arc4random();
initstate(seed, state, 32);
long something_something = random(); // This and subsequent calls to random will use the initialized state

[glebius]

In D43775#998621, @asomers wrote:

Oh, I'm not suggesting that you run the test with a fixed seed every time. I'm suggesting that you run it with a different random seed every time, but print the seed. That way, a user can quickly reproduce the failure by modifying the test to use the fixed seed instead. Like this:

That's a good idea, I will try to implement it.

[glebius]

Use nrand48() with random seed and print seed on failure.

[glebius]

• fix false failure when random engine returns us 0, that makes records[n] == records[n + 1]
• improve printing of seed in case of failure, so that can be easily pasted into initializer

[asomers]

This looks good. BTW, you don't have to wait until the cleanup phase to print the seed. It's OK to print it at the beginning of the test phase. In the event of a failure, Kyua will report everything that the test printed.

[glebius]

Print seed always, rely on kyua to hide it when test passes. Simplifies a lot.

[rlibby]

@glebius I ran into this while running the kern tests. This new test is failing reliably for me, and it looks like in CI too, e.g. this run on April 9:
https://ci.freebsd.org/job/FreeBSD-main-amd64-test/25066/testReport/sys.kern/unix_seqpacket_test/random_eor_and_waitall/

It fails for me every time with this, on an amd64 GENERIC vm:
kyua debug -k /usr/tests/sys/Kyuafile kern/unix_seqpacket_test:random_eor_and_waitall

I've seen it fail in a few different ways:

% for i in {1..10}; do kyua debug -k /usr/tests/sys/Kyuafile kern/unix_seqpacket_test:random_eor_and_waitall; done
Using seed: 0x41fd, 0xd11e, 0x7725, 0xadf8, 0xe04f, 0x1d61,
*** Check failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1255: len != iov.iov_len: recvmsg(MSG_WAITALL): 1132, expected 3141
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x8fa8, 0xdbe5, 0x1403, 0xb14d, 0x84f8, 0xfbd0,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x5b73, 0xc363, 0x39d7, 0xc52a, 0xfa9d, 0x15ab,
*** Check failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1255: len != iov.iov_len: recvmsg(MSG_WAITALL): 4484, expected 27917
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1269: data corruption past 4923
Using seed: 0xaa47, 0x3831, 0xb603, 0x97df, 0xb839, 0x0109,
*** Check failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1255: len != iov.iov_len: recvmsg(MSG_WAITALL): 4525, expected 9299
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x679a, 0xc263, 0xa25f, 0x348c, 0x2d3a, 0x0cd2,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0xaa1e, 0x7317, 0x2dde, 0xe299, 0x1139, 0xf8d8,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x6b58, 0x2247, 0x5d93, 0x9c57, 0x326d, 0x1614,
*** Check failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1255: len != iov.iov_len: recvmsg(MSG_WAITALL): 682, expected 24997
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x8eca, 0xc5bd, 0xc09d, 0xe15e, 0xe7c3, 0xfdad,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x2859, 0x311b, 0x69d4, 0xd44c, 0xce3d, 0xe01b,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met
Using seed: 0x0329, 0x992f, 0x6937, 0x766c, 0x47e5, 0x5270,
kern/unix_seqpacket_test:random_eor_and_waitall  ->  failed: /usr/src/freebsd/tests/sys/kern/unix_seqpacket_test.c:1182: send(params->sock, &params->sendbuf[off], len, flags) == len not met

I traced the the errno for the send() failure as EMSGSIZE.