Right now this is a continuation of D29691, I decided to not commandeer the original review.
I switched to a mode where only one thread is temporary run, only to dump core, instead of unsuspending everything, and then trying to catch them in AST. So the process is kept stopped after PT_COREDUMP, but debugger needs to wait on it before PT_CONTINUE is allowed.
I am sure that this requires much more thinking and testing, but I want to provide you with something to not block any other work.
Also I disallowed parallel ptrace(2) operations, so that PT_DETACH cannot occur while PT_COREDUMP is running.
I also tried to plug as many holes in the patch as possible, non-complete list is
- usermode flags
- added padding for future ABI-compat extensions
- minor bugs