Page MenuHomeFreeBSD

Rewrite pfctl_test in C to reduce testsuite run time
ClosedPublic

Authored by arichardson on Oct 14 2020, 2:08 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mar 7 2024, 2:58 PM
Unknown Object (File)
Feb 8 2024, 2:44 PM
Unknown Object (File)
Dec 20 2023, 5:24 AM
Unknown Object (File)
Nov 25 2023, 8:25 AM
Unknown Object (File)
Nov 14 2023, 6:38 PM
Unknown Object (File)
Nov 9 2023, 4:04 AM
Unknown Object (File)
Nov 8 2023, 6:59 PM
Unknown Object (File)
Nov 8 2023, 12:56 PM
Subscribers

Details

Summary

The new C test takes 25 seconds on QEMU-RISC-V, wheras the shell version
takes 332 seconds.

Even with the latest optimizations to atf-sh this test still takes a few
seconds to startup in QEMU. Re-writing it in C reduces the runtime for a
single test from about 2-3 seconds to less than .5 seconds. Since there
are ~80 tests, this adds up to about 3-4 minutes.
This may not seem like a big speedup, but before the recent optimizations
to avoid atf_get_srcdir, each test took almost 100 seconds on QEMU RISC-V
instead of 3. This also significantly reduces the time it takes to list
the available test cases, which speeds up running the tests via kyua:

root@qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test_old
...
158/158 passed (0 failed)
      332.08 real        42.58 user       286.17 sys
root@qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test
158/158 passed (0 failed)
       24.96 real         9.75 user        14.26 sys

root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001
pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in
---
binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64
binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64
---
passed
        0.17 real         0.06 user         0.08 sys
root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old pf1001
pfctl_test_old: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test_old: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Id  Refs Name
141    1 pf
Executing command [ pfctl -o none -nvf - ]
passed
        1.73 real         0.25 user         1.41 sys
root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old -l > /dev/null
       24.36 real         2.26 user        21.86 sys
root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null
        0.04 real         0.02 user         0.01 sys

The speedups are even more noticeable on CHERI-RISC-V (since QEMU runs
slower when emulating CHERI instructions):

root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new -l > /dev/null
        0.51 real         0.49 user         0.00 sys
root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null
       34.20 real        32.69 user         0.16 sys
root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001
pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Id  Refs Name
147    1 pf
Executing command [ pfctl -o none -nvf - ]
passed
        5.74 real         5.41 user         0.03 sys
root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new pf1001
pfctl_test_new: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test_new: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in
---
binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64
binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64
---
passed
        0.68 real         0.66 user         0.00 sys
root@qemu-cheri-alex:/usr/tests/sbin/pfctl #
Test Plan

The new test passes.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

arichardson created this revision.

I've not yet looked into the why, but this seems to fail for me:

pfctl_test:pf0001 -> failed: /usr/src/sbin/pfctl/tests/pfctl_test.c:100: Expected errno 0, got 2, in count == 0 && "Should have reached EOF" [0.007s]

In D26779#597070, @kp wrote:

I've not yet looked into the why, but this seems to fail for me:

pfctl_test:pf0001 -> failed: /usr/src/sbin/pfctl/tests/pfctl_test.c:100: Expected errno 0, got 2, in count == 0 && "Should have reached EOF" [0.007s]

Hmm that's strange, that would be ENOENT. What is your kyua command line?

Simple /usr/tests/sbin/pfctl % sudo kyua test (So I start kyua in the /usr/tests/sbin/pfctl directory)

  • set errno to zero before read() loop
In D26779#597081, @kp wrote:

Simple /usr/tests/sbin/pfctl % sudo kyua test (So I start kyua in the /usr/tests/sbin/pfctl directory)

I can't reproduce this locally, but maybe the udpated patch helps?

Yeah, the errno made the difference. The tests pass now (and quite a lot faster than before).

I feel sure you meant to spell pfct_test_list.inc as pfctl_test_list.inc

Other than the file name nit I think this can go in.

This revision is now accepted and ready to land.Oct 14 2020, 5:16 PM
This revision now requires review to proceed.Oct 14 2020, 5:33 PM
This revision was not accepted when it landed; it landed in state Needs Review.Oct 14 2020, 5:40 PM
This revision was automatically updated to reflect the committed changes.