You commented in IRC that you didn't believe the NOTE_REAP was being delivered. Here's what I did:

I deployed these changes and then changed the rc.subr wait_for_pids to add -v to pwait and I can see the "collected" message in the output now

root@current:/usr/src/bin/pwait # service gitea restart
Stopping gitea.
Waiting for PIDS: 72212, 72212: exited with status 0. 72212: collected..

sometimes I'm seeing two NOTE_EXIT events come through like this:

# service gitea restart
Stopping gitea.
Waiting for PIDS: 71485, 71485: exited with status 0. 71485: exited with status 0. 71485: collected..

That is probably from the immediate delivery of the NOTE_EXIT because the processed was flagged as exiting, but the other delivery of NOTE_EXIT in kern_exit.c didn't happen yet.

Moving the discussion from a mail, there:

On Sun, Jul 19, 2026 at 12:48:35PM +0200, Dag-Erling Smørgrav wrote:

A while ago, I tried adding a NOTE_REAP which triggers when a process is
reaped. With debug logging added, I can see that the event gets
generated at the correct point, but for reasons I was unable to figure
out at the time, it never gets delivered.

Perhaps because knotes are dropped on the NOTE_EXIT:

if (event == NOTE_EXIT) {
        kn->kn_flags |= EV_EOF | EV_ONESHOT;

and EV_ONESHOT causes the knote_drop() in kqueue_scan() on reporting.

The patch is here: https://reviews.freebsd.org/D58313

A usage example is here: https://reviews.freebsd.org/D58314

I would really like to get this sorted out before 14.5, because
rc.subr's wait_for_pids() is currently broken in all branches.

Can you explain in which sense it is broken, and what is the intent of the NOTE_REAP?

There are more operations to clear the process, besides reaping. For instance, freeing of the PID might occur after reaping.
There is a planned change that (IMO significantly) modifies how the reaping is handled, see D58264.
I suspect it could interact with what you are doing there.

In D58314#1338061, @kib wrote:

Can you explain in which sense it is broken, and what is the intent of the NOTE_REAP?

pwait currently subscribes to NOTE_EXIT and gets the event too early, so when rc.subr is starting up a service you restarted it checks if the process is already running and finds it in the process table and throws an error. For many services it is not possible to "service foo restart" successfully on FreeBSD 14.4-RELEASE or 15.1-RELEASE. If you try to restart services as fast as possible it's quite easy to hit the race condition and see the error. A few people have noticed that software written in Go is unusually susceptible to this problem.

In D58314#1338061, @kib wrote:

A while ago, I tried adding a NOTE_REAP which triggers when a process is
reaped. With debug logging added, I can see that the event gets
generated at the correct point, but for reasons I was unable to figure
out at the time, it never gets delivered.

Perhaps because knotes are dropped on the NOTE_EXIT:

if (event == NOTE_EXIT) {
        kn->kn_flags |= EV_EOF | EV_ONESHOT;

and EV_ONESHOT causes the knote_drop() in kqueue_scan() on reporting.

No. Please look at the actual patch (D58313) instead of speculating:

The patch is here: https://reviews.freebsd.org/D58313

A usage example is here: https://reviews.freebsd.org/D58314

and again, I know the event gets generated, I instrumented kqueue_scan(), but it's never delivered to the process.

This obviously changes the semantic of the pwait(1) tool. Before, it returned control after the process exited. Now, it additionally wait until a reaper makes the wait*() call.

It might be that the new behavior is fine if directed by an option, otherwise I do not think that this is acceptable.

In D58314#1339356, @kib wrote:

This obviously changes the semantic of the pwait(1) tool. Before, it returned control after the process exited. Now, it additionally wait until a reaper makes the wait*() call.

But what does "exited" really mean in this context?

What users are seeing is something like "service gitea restart" fails to start the process again. pwait believed that the process exited (NOTE_EXIT received), but in reality it's still in the process table with the ps state column = REJ (runnable, trying to exit).

Do we know what could have changed in the kernel that surfaced this problem in the first place? I don't remember encountering this with 14.3-RELEASE and older, but it's easily reproduced with 14.4 and 15.0+.

It looks like the process is still exiting, but has not successfully exited.

Thanks for the additional eyes on this.

In D58314#1339359, @feld wrote:

In D58314#1339356, @kib wrote:

This obviously changes the semantic of the pwait(1) tool. Before, it returned control after the process exited. Now, it additionally wait until a reaper makes the wait*() call.

But what does "exited" really mean in this context?

Exited means that the process developed an exit status, available for retrieval by several means. Additional guarantee is that no process thread is ever going to execute anything in user space.

What users are seeing is something like "service gitea restart" fails to start the process again. pwait believed that the process exited (NOTE_EXIT received), but in reality it's still in the process table with the ps state column = REJ (runnable, trying to exit).

Do we know what could have changed in the kernel that surfaced this problem in the first place? I don't remember encountering this with 14.3-RELEASE and older, but it's easily reproduced with 14.4 and 15.0+.

Nothing changed.

It looks like the process is still exiting, but has not successfully exited.

Thanks for the additional eyes on this.

In D58314#1339359, @feld wrote:

Do we know what could have changed in the kernel that surfaced this problem in the first place? I don't remember encountering this with 14.3-RELEASE and older, but it's easily reproduced with 14.4 and 15.0+.

Nothing changed in the kernel. I started out fixing bugs in /etc/rc.subr's wait_for_pids(), then tried to make it more efficient, and ended up making it _too_ efficient, to the point were services commonly try to restart after the previous instance has exited but before the process has been reaped. Most daemons either clear their pidfile on exit or don't care, but some don't clear the pidfile on exit and check the pid that it contains on startup, and refuse to start because you can still kill -0 the zombie.

In D58314#1339356, @kib wrote:

It might be that the new behavior is fine if directed by an option, otherwise I do not think that this is acceptable.

Which option character would you suggest? I started out with -z (for “zombie”), but perhaps -r (for “reaped”) is more appropriate?

In D58314#1339370, @des wrote:

Nothing changed in the kernel. I started out fixing bugs in /etc/rc.subr's wait_for_pids(), then tried to make it more efficient, and ended up making it _too_ efficient

Thank you, now it all makes sense.