The SysV message queue code has the same problem that we fixed in the
semaphore code in SA-26:54.sysvsem and SA-26:64.sysvsem. That is,
kern_msgrcv() and kern_msgsnd() are susceptible to TOCTOU races when
they sleep waiting for messages and queue space, respectively. Unlike
the semaphore code, however, the message queue array is never resized
and there is no notion of message queue sets, so such races do not open
up any memory safety bugs.
Fix the races, following the pattern from the SysV semaphore changes:
use a global sequence number array with wide counters, to avoid changing
the KBI which is embedded in userspace tools. Fix kern_msgsnd(), which
did not validate the sequence number at all after sleeping. Use a
msgvalid() helper to factor out some duplicated checks. Convert a
couple of local flag variables to bools.
Reported by: Apex Security Research