I got a watchdog timeout with this patch: https://people.freebsd.org/~pho/stress/log/log0013.txt

FWIW, I upload patch series as a set of arc reviews. Originally this was tedious (using arc diff --create --head <N> <N-1> for each commit), but @markj has been working on a script that automates this (I used it to upload the series of zombproc-related debugger cleanups as a single command from a branch with multiple commits).

20210101 07:17:42 all (6/740): machipc2.sh
panic: Lock (sx) proctree not locked @ kern/kern_proc.c:1172.
cpuid = 17
time = 1609481864
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01c7c16ab0
vpanic() at vpanic+0x181/frame 0xfffffe01c7c16b00
panic() at panic+0x43/frame 0xfffffe01c7c16b60
witness_assert() at witness_assert+0x23a/frame 0xfffffe01c7c16b90
fill_kinfo_proc() at fill_kinfo_proc+0x4e/frame 0xfffffe01c7c16be0
kern_proc_out() at kern_proc_out+0x62/frame 0xfffffe01c7c17370
sysctl_out_proc() at sysctl_out_proc+0x68/frame 0xfffffe01c7c17840
sysctl_kern_proc() at sysctl_kern_proc+0x87/frame 0xfffffe01c7c17890
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame 0xfffffe01c7c178e0
sysctl_root() at sysctl_root+0x20d/frame 0xfffffe01c7c17960
userland_sysctl() at userland_sysctl+0x180/frame 0xfffffe01c7c17a10
sys___sysctl() at sys___sysctl+0x5f/frame 0xfffffe01c7c17ac0
amd64_syscall() at amd64_syscall+0x147/frame 0xfffffe01c7c17bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01c7c17bf0
--- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x8003fb9ea, rsp = 0x7fffffffd258, rbp = 0x7fffffffd290 ---
KDB: enter: panic
[ thread pid 78475 tid 792240 ]
Stopped at      kdb_enter+0x37: movq    $0,0x10960c6(%rip)
db>

https://people.freebsd.org/~pho/stress/log/log0014.txt

I realize this is quite old, but I'd like to discuss the possibility of reverting 5844bd058aed, as it appears to have a significant performance impact which was never discussed at the time, while the benefit of the change appears to be purely hypothetical. I have no reason to doubt that pgrp_calc_jobc() is correct, but my own attempts to investigate possible errors in tracking pg_jobc did not uncover anything except an unrelated race condition in ppoll() and pselect() (cf. D47738 and D47741). I know of historical instances, e.g. D26116, but see no evidence of any unresolved issues at the time. Meanwhile, the cost of pgrp_calc_jobc() is quite high, especially in a debugging kernel where the process tree lock is being asserted at least four times per iteration of the inner loop, and it is being incurred even when no change occurs, e.g. on every ps or procstat -a or refresh of top.

In D27871#1101805, @des wrote:

I realize this is quite old, but I'd like to discuss the possibility of reverting 5844bd058aed, as it appears to have a significant performance impact which was never discussed at the time, while the benefit of the change appears to be purely hypothetical. I have no reason to doubt that pgrp_calc_jobc() is correct, but my own attempts to investigate possible errors in tracking pg_jobc did not uncover anything except an unrelated race condition in ppoll() and pselect() (cf. D47738 and D47741). I know of historical instances, e.g. D26116, but see no evidence of any unresolved issues at the time. Meanwhile, the cost of pgrp_calc_jobc() is quite high, especially in a debugging kernel where the process tree lock is being asserted at least four times per iteration of the inner loop, and it is being incurred even when no change occurs, e.g. on every ps or procstat -a or refresh of top.

I believe I gave up on trying to maintain counters by increments after jhb' report of failures (assertions were triggered) on some gdb tests. I.e. it should be added orphanage and exit process reparenting to the tests to trigger. I do not remember details, only that after several days of trying to handle it, I decided to recalculate the state instead of adjusting it.

Where do you see the issues with performance? Just in the assertion times, like sx_assert (due to WITNESS, perhaps)? Or is it the actual LIST_FOREACH() iterations in the pgrp_calc_jobc()?