ptrace: Refresh tracing state after syscall execution

[mchoo]

Summary

syscallenter() samples P_TRACED before fetching arguments and invoking the
syscall handler. A debugger can attach or detach while that handler runs,
so the entry-time snapshot does not describe the tracing state when the
handler returns.

Re-evaluate traced at the common retval path, including early error
returns. Check TDB_SCE, TDB_BOUNDARY and TDB_SET_SC_RET so that flags
left by a debugger that detached are still cleared.

MFC after: 2 weeks
Sponsored by: FreeBSD Foundation

[kib]

sys/kern/subr_syscall.c

212

But why re-reading td_dbgflags? If debugger is not attached, the flags should not matter.
It might be reasonable to clear them for all threads on attach.

I am trying to avoid adding any additional memory accesses, because they are observable on the nano-benchmarks like tools/tools/syscall_timing. From this PoV, re-fetching p_flag once is a necessary evil, but additional fetches can be guarded by P_TRACED.

[mchoo]

sys/kern/subr_syscall.c

212

(td->td_dbgflags & (TDB_SCE |
	    TDB_BOUNDARY | TDB_SET_SC_RET))

is not needed when it is cleared on attach.

But reading and comparing to TDB_USERREGS in the original patch is required because if a debugger is detached during sys call and has set registers before, we need to preserve the modified registers after syscall. That wouldn't be necessary if are allowed to ignore PT_SETREGS if debugger was detached before syscall returned, but I don't think that's what it should be.