Page MenuHomeFreeBSD

tests: Add Linuxulator regression test infrastructure
Needs ReviewPublic

Authored by vova_fbsd.ru on Tue, Oct 13, 2:57 PM.
Tags
None
Referenced Files
F176434559: D60702.id190046.diff
Sun, Oct 18, 9:09 PM
F176434552: D60702.id190156.diff
Sun, Oct 18, 9:09 PM
F176423310: D60702.diff
Sun, Oct 18, 7:34 PM
F176390903: D60702.id190156.diff
Sun, Oct 18, 12:12 PM
F176384057: D60702.id190046.diff
Sun, Oct 18, 10:38 AM
Unknown Object (File)
Sat, Oct 17, 9:42 PM
Unknown Object (File)
Sat, Oct 17, 5:46 AM
Unknown Object (File)
Sat, Oct 17, 4:59 AM
Subscribers

Details

Reviewers
glebius
ngie
igoro
olivier
dchagin
kib
Group Reviewers
tests
Summary

Add infrastructure for running regression tests whose test subjects must
execute under the Linux ABI.

Keep the test sources and the reusable Linux-side runtime under
sys/tests/linux. Add tests/sys/linux as the build-system integration
point and register it in the system test hierarchy, installing the tests
under /usr/tests/sys/linux.

Linux test programs are built as static, freestanding ELF binaries. They
do not depend on a Linux base system, libc, an ELF interpreter, or any
files below compat.linux.emul_path. This allows the tests to run on
minimal test systems and in buildworld test sandboxes.

Provide a small reusable runtime for Linux-side test helpers:

  • linux_crt0 supplies the ELF entry point, invokes a conventional main(), and terminates the process with the Linux exit_group system call.
  • linux_syscalls provides the architecture-specific raw system call interface and a minimal set of syscall wrappers.
  • amd64 and arm64 are supported, producing x86-64 and AArch64 Linux ELF executables respectively.

Linux kernel modules required by individual tests are declared through
the ATF require.kmods metadata. Kyua can therefore load linux64.ko before
executing a test, without requiring each test to implement module
management.

Add the first regression test for Linux controlling-terminal semantics.
The native ATF test creates a pseudo-terminal, establishes it as the
controlling terminal of a new session, and executes a static Linux
helper. The helper closes the last descriptors referencing the slave
and then attempts to reopen /dev/tty.

Linux keeps the session's controlling-terminal reference across the last
slave close, so reopening /dev/tty must succeed. This exercises the
devfs behavior fixed by D60701 ("devfs: Keep a Linux controlling
tty across the last slave close").

Also register /usr/tests/sys/linux in BSD.tests.dist so that make check
creates the directory in its temporary installation hierarchy.

Test Plan
# make -C tests/sys/linux check
...
devfs_pty:ctty_last_close  ->  passed  [0.003s]

Results file id is usr_obj_usr_src_arm64.aarch64_tests_sys_linux_checkdir_usr_tests_sys_linux.20261013-152246-721666
Results saved to /root/.kyua/store/results.usr_obj_usr_src_arm64.aarch64_tests_sys_linux_checkdir_usr_tests_sys_linux.20261013-152246-721666.db

1/1 passed (0 broken, 0 failed, 0 skipped)

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 78160
Build 75043: arc lint + arc unit

Event Timeline

vova_fbsd.ru edited the test plan for this revision. (Show Details)
vova_fbsd.ru edited the test plan for this revision. (Show Details)

Thanks a lot for this project! It is definitely very valuable. Adding more reviewers.

sys/tests/linux/Makefile
21–25

As more tests are added you definitely don't want to copy over all these CFLAGS. I don't know what is better, to append CFLAGS or to fully override them.

vova_fbsd.ru edited the test plan for this revision. (Show Details)

tests: Consolidate Linux helper build flags and brand binaries

Define the compiler and linker flags used for static Linux test
helpers in one place and use a common rule to build all helpers.

Explicitly brand the resulting binaries as Linux. Clang and lld
otherwise produce static ELF binaries with the System V OSABI and no
ABI note, which only worked when kern.elf64.fallback_brand was set to
Linux.

This makes the tests independent of the host's fallback ELF brand
configuration and simplifies adding more Linux helpers.

Thank you for this, but I'm wondering if the functional testing for Linux syscalls are better suited for LTP. I'm not as active with that project as I used to be, but I'm still able to contribute there. There is some support missing from FreeBSD that would need to be added in order to make those tests work (/dev/loop), and some of the tests assume explicit Linux support which FreeBSD lacks (SCHED_EXT, etc), but there might be some items that can be gleaned from that project that allows us to avoid reinventing the wheel too much.
Treating this as a simple smoke test though for Linuxulator would be ok, but I want to make sure the tests are constrained to smoke tests and we don't go reinventing the wheel too much.

sys/tests/linux/devfs_pty_linux.c
14–16

Please use linux_close_range instead.

sys/tests/linux/linux_crt0.c
17

Who is expected to define __x86_64__? FreeBSD has typically used __amd64__, not __x86_64__ (this is a Linux-ism).

% rg -c __x86_64__ | awk -F: 'NF == 2 { s += $2 } END { print s }'
2050
% rg -c __amd64__ | awk -F: 'NF == 2 { s += $2 } END { print s }'
2389
%

Most of the __x86_64__ uses are in contrib.

18–33

Where did the assembly here come from? FreeBSD source? GNU/Linux source?

sys/tests/linux/linux_crt0.h
1–12

Is this header necessary?

sys/tests/linux/linux_syscalls.c
9
22–47

These are architecture specific syscall numbers. Is there a way that a symbolic constant like __NR_open (the Linux-documented symbol), etc, could be used instead? Per syscall(2), they should be available in sys/syscall.h

54

What about syscalls that accept more or less than 4 arguments?

sys/tests/linux/linux_syscalls.h
10

This should be defined in the non-test linux_syscalls.h header.

12–15

These should use the Linux(-ish) types so the syscalls compile properly on 32-bit and 64-bit.

tests/sys/linux/Makefile
1–3
  • Why the indirection?
  • How can a user specify that they don't want to try and build and install the tests?
  • What about unsupported architectures, e.g., powerpc64?

Who is expected to define x86_64? FreeBSD has typically used amd64, not x86_64 (this is a Linux-ism).

I assume, it is defined by --target=x86_64-linux-gnu but I agree, FreeBSD-style will be better

I'm wondering if the functional testing for Linux syscalls are better suited for LTP

I would assime that LTP tests - are good for extended set of tests that require to setup additional software packages (linux base in that case).
As far I understand we have no such beast yet in freebsd-src, so all tests should be buildable/runnable with just base.
Intent was to cover with tests linxlator fixes. I would assume that running whole LTP would be a little challenging, but good to have.

Where did the assembly here come from? FreeBSD source? GNU/Linux source?

This assembly was written specifically for these freestanding test helpers; it was not copied from FreeBSD or GNU/Linux sources. It provides only the minimum ABI entry point needed to align/initialize the call frame and invoke linux_start(). The helpers do not currently consume argc, argv, envp, or the auxiliary vector.

sys/tests/linux/Makefile
17

I think you need something like -ffreestanding otherwise compiler is allowed to assume presence of some STD C runtime.

sys/tests/linux/devfs_pty_linux.c
12

I think the type should be int.

tests: Refine Linuxulator test infrastructure

Move the Linuxulator userland tests from sys/tests/linux to
tests/sys/linux. sys/tests is intended for kernel-side test modules,
while ATF tests and their helper programs belong under tests/sys.

Use the generated architecture-specific Linux syscall numbers instead
of embedding raw syscall numbers in the test support code. Extend the
low-level syscall primitive to pass all six Linux ABI arguments on
amd64 and arm64.

Use close_range(2) to close the standard descriptors in the devfs PTY
helper, remove the unnecessary crt0 header, and use the FreeBSD
amd64 spelling for the architecture check.

Only descend into tests/sys/linux on architectures for which the
freestanding Linux helpers can currently be built.

Suggested by: ngie
Differential Revision: https://reviews.freebsd.org/D60702

sys/tests/linux/Makefile
17

I think you need something like -ffreestanding otherwise compiler is allowed to assume presence of some STD C runtime.

If I am not mistaken without -ffreestanding optimizer may "replace" memory() and simiar function with optimized implementation, which is fine, even useful. But it will not build with libc.a

sys/tests/linux/devfs_pty_linux.c
12

I think the type should be int.

well, it is raw syscall result, to not overcomplicate wrapper (normal libc handles conversion)

sys/tests/linux/Makefile
17

For hosted compilation environment, a compiler can assume the presence of all facilities described in the C standard. For freestanding, it is indeed just some mem* functions, basically.

tests: Check native controlling terminal release on last close

Add a regression test verifying the native FreeBSD controlling terminal
semantics.

Create a new session, acquire a pseudo-terminal slave as the controlling
terminal, and close the last descriptor referencing the slave. Verify
that a subsequent open of /dev/tty fails with ENXIO.

Use pipes to synchronize the parent and child so that the tested close is
guaranteed to be the final slave close.

This complements the Linuxulator test, where the controlling terminal is
expected to persist across the last close.

How can a user specify that they don't want to try and build and install the tests?

Now, when all these tests under tests/sys/linux - user can control what to build using usual approach as with all other tests

What about unsupported architectures, e.g., powerpc64?

Nothing will be built or run as whole directory will be skipped as

TESTS_SUBDIRS+=         ${_linux}

.if ${MACHINE_ARCH} == "amd64" || ${MACHINE_ARCH} == "aarch64"
_linux= linux
.endif

I'm wondering if the functional testing for Linux syscalls are better suited for LTP.

as for LTP ... well it worth to have port to run relevant part of LTP, but I would assume that should be a standalone initiave
I've run it just for curiocity:

Target information
──────────────────
Kernel:   Linux 5.15.0 FreeBSD 16.0-CURRENT #3 main-n290167-43c79c64af48-dirty: Fri Oct
Cmdline:  BOOT_IMAGE=/boot/kernel/kernel
          ro
          root=302
Machine:  aarch64
Arch:     aarch64
RAM:      4156508 kB
Swap:     2097152 kB
Distro:   rocky 9.8

────────────────────────
      TEST SUMMARY
────────────────────────
Suite:   syscalls
Runtime: 37m 44s
Runs:    1551

Results:
    Passed:   4520
    Failed:   3762
    Broken:   1544
    Skipped:  624
    Warnings: 33

it is defenetely not ready to be used as green/red indicator for FreeBSD kernel regressions,

tests: Add Linux epoll descriptor inheritance regression test

Add a Linux ABI regression test for preserving epoll descriptors across
fork. This covers the behavior implemented by 226616ad7dba ("linux:
preserve epoll descriptors across fork using KQUEUE_CPONFORK").

Build the test subject as a static, freestanding Linux ELF executable,
using the common Linux test runtime. A native ATF wrapper executes the
helper under Linuxulator and reports its exit status through Kyua. No
Linux base system or Linux libc is required.

Exercise epoll instances created both with and without EPOLL_CLOEXEC.
For each variant, verify in the child process that:

  • the epoll descriptor remains valid after fork;
  • F_GETFD reports the expected FD_CLOEXEC state;
  • an ordinary pipe descriptor is inherited as a control case;
  • a ready registration and its event data are available;
  • epoll_ctl(EPOLL_CTL_MOD) succeeds after allocating additional file descriptors.

The final check reproduces the descriptor-reuse failure mode seen before
the fix. When the epoll descriptor was omitted from the copied file
descriptor table, a newly created pipe could reuse its number, causing
subsequent epoll_ctl calls to fail with EINVAL.

Limit the test to the semantics provided by KQUEUE_CPONFORK. It does not
test shared epoll state between parent and child: registration changes
and one-shot consumption are still independent because the underlying
kqueue is copied rather than shared.

Extend the reusable freestanding Linux syscall layer with the fork,
wait4, pipe, fcntl, and epoll operations needed by this test. Provide
the appropriate syscall implementations for Linux x86-64 and AArch64.

The test passes on FreeBSD 16-CURRENT arm64 with linux64.ko.

tests/sys/linux/epoll_fork_linux.c
22 ↗(On Diff #190156)

What does -4 represent here: why isn't this a constant?

tests/sys/linux/linux_syscalls.c
1 ↗(On Diff #190156)

This seems like it's going to get difficult to manage as more architecture support is added.

27 ↗(On Diff #190156)

long isn't the right return type for many of these syscalls. Please fix this to use the correct documented types: https://linux.die.net/man/2/

tests/sys/linux/linux_syscalls.h
37–50 ↗(On Diff #190156)

These seem like they should already be defined by the linuxulator sysent-produced headers. Is there any way to incorporate them instead of duplicating the prototypes here?

tests: Generate Linux syscall stubs for Linuxulator tests

Generate freestanding Linux syscall stubs directly from the Linuxulator
syscalls.master files instead of maintaining architecture-specific syscall
number mappings and C wrappers.

Emit stubs for all native syscalls and place each one in a separate ELF
section. Link the test helpers with --gc-sections so that only referenced
stubs are retained.

Use syscalls available on both amd64 and arm64 directly, including openat,
pipe2, clone, epoll_pwait, wait4, and exit_group. This removes the
architecture-specific compatibility wrappers and linux_syscalls.c.

Give the generated stubs libc-like error handling: translate raw Linux
error returns to -1 and store the positive error number in errno. The
test helpers are single-threaded, so a non-TLS errno is sufficient.

Use the existing Linuxulator event and errno definitions where their
headers are self-contained, and retain local definitions only for ABI
constants whose kernel headers cannot be included by freestanding
userspace code.

Tested on FreeBSD/aarch64 with:

make -C /usr/src/tests/sys/linux clean all check

Both devfs_pty and epoll_fork pass. The Linux helper binaries also
cross-build successfully for amd64.