pwait: Don't exit until process is reaped

[des]

Summary

In addition to NOTE_EXIT, use NOTE_REAP to be notified when the target
process is reaped.

PR: 293183
MFC after: 1 week

[feld]

You commented in IRC that you didn't believe the NOTE_REAP was being delivered. Here's what I did:

I deployed these changes and then changed the rc.subr wait_for_pids to add -v to pwait and I can see the "collected" message in the output now

root@current:/usr/src/bin/pwait # service gitea restart
Stopping gitea.
Waiting for PIDS: 72212, 72212: exited with status 0. 72212: collected..

sometimes I'm seeing two NOTE_EXIT events come through like this:

# service gitea restart
Stopping gitea.
Waiting for PIDS: 71485, 71485: exited with status 0. 71485: exited with status 0. 71485: collected..

That is probably from the immediate delivery of the NOTE_EXIT because the processed was flagged as exiting, but the other delivery of NOTE_EXIT in kern_exit.c didn't happen yet.

[kib]

Moving the discussion from a mail, there:

On Sun, Jul 19, 2026 at 12:48:35PM +0200, Dag-Erling Smørgrav wrote:

A while ago, I tried adding a NOTE_REAP which triggers when a process is
reaped. With debug logging added, I can see that the event gets
generated at the correct point, but for reasons I was unable to figure
out at the time, it never gets delivered.

Perhaps because knotes are dropped on the NOTE_EXIT:

if (event == NOTE_EXIT) {
        kn->kn_flags |= EV_EOF | EV_ONESHOT;

and EV_ONESHOT causes the knote_drop() in kqueue_scan() on reporting.

The patch is here: https://reviews.freebsd.org/D58313

A usage example is here: https://reviews.freebsd.org/D58314

I would really like to get this sorted out before 14.5, because
rc.subr's wait_for_pids() is currently broken in all branches.

Can you explain in which sense it is broken, and what is the intent of the NOTE_REAP?

There are more operations to clear the process, besides reaping. For instance, freeing of the PID might occur after reaping.
There is a planned change that (IMO significantly) modifies how the reaping is handled, see D58264.
I suspect it could interact with what you are doing there.

[feld]

In D58314#1338061, @kib wrote:

Can you explain in which sense it is broken, and what is the intent of the NOTE_REAP?

pwait currently subscribes to NOTE_EXIT and gets the event too early, so when rc.subr is starting up a service you restarted it checks if the process is already running and finds it in the process table and throws an error. For many services it is not possible to "service foo restart" successfully on FreeBSD 14.4-RELEASE or 15.1-RELEASE. If you try to restart services as fast as possible it's quite easy to hit the race condition and see the error. A few people have noticed that software written in Go is unusually susceptible to this problem.