ptmx_fdopen() masks the supplied file flags to FREAD | FWRITE before
passing them to pts_alloc(). This discards FNONBLOCK even when the
caller explicitly requests O_NONBLOCK.
As a result, open("/dev/ptmx", O_RDWR | O_NOCTTY | O_NONBLOCK) succeeds,
but the returned file is blocking. F_GETFL reports O_NONBLOCK cleared,
and a subsequent read can block instead of returning EAGAIN when no
data is available.
This causes Linux xfce4-terminal/VTE running under Linuxulator to
freeze. VTE opens the PTY master with O_NONBLOCK and relies on
nonblocking reads while draining terminal output. Once the available
data has been consumed, the main UI thread blocks in ptsdev_read(),
leaving the window unresponsive even though the child shell has
started successfully.
Include FNONBLOCK in the flags passed to pts_alloc(). Blocking opens
retain their existing behavior. No change to posix_openpt(2) flag
validation is required.
The flag loss also reproduces with a native FreeBSD program, so the
problem is in the shared /dev/ptmx implementation rather than Linux
syscall flag translation.
Validation:
- Before the fix, a native FreeBSD reproducer observes O_NONBLOCK cleared after opening /dev/ptmx with that flag.
- Explicitly setting O_NONBLOCK with F_SETFL works, confirming that nonblocking PTY operation itself is supported.
- After the fix, F_GETFL preserves O_NONBLOCK when requested and leaves it cleared for a blocking open.
- xfce4-terminal in the Linux jail displays the shell prompt and accepts input.