malloc debugging: detect mismatched mtp in alloc/free with redzone

[rlibby]

Summary

XXX This is just a quick hack / demo. It would be nice to have
debugging for mismatched malloc types between alloc/free. A simple way
is to stash the mtp pointer alongside the malloc memory. This hacks
redzone to do that, because it already stashes stuff alongside the
malloc memory, has the pointer adjustments for doing so, stores a stack,
etc. I am hoping for review of the general idea, and of doing it with
redzone or not. Any better way?

---

Mismatched mtp in malloc/free causes malloc accounting bugs. Enhance
redzone so that it stashes a context cookie (the mtp pointer) on alloc
and checks it on free.

This is a bit of a misuse of redzone(9), but the machinery is already
available.

Test Plan

Manually insert a bug into accept1() and observe that this detects it.

[markj]

So while it seems perfectly fine to use redzone for this purpose, I suspect it'd be somewhat ineffective for debugging since users typically do not run with redzone configured.

Have you considered including DEBUG_REDZONE in INVARIANTS kernel configurations? Then, provide a tunable to enable it. That way you could at least try and catch the bug without having to recompile the kernel. Ideally you'd be able to flip redzone on with a sysctl but I can't really see a way to implement that since redzone_check() has no way of knowing whether a given chunk of memory was padded. You might even implement multiple modes for redzone: one, perhaps on by default, pads with only enough space to do mtp checking, and the other does full checking, stack-saving, etc..

I suspect there's no good reason to exclude DEBUG_REDZONE (and maybe even MEMGUARD on platforms with plentiful KVA) from INVARIANTS kernels now that we have GENERIC-NODEBUG.