random: Change the entropy harvest event queuing scheme

The entropy queue stores entropy gathered from environmental sources.
Periodically (every 100ms currently), the random kthread will drain this
queue and mix it into the CSPRNG's entropy pool(s).

The old scheme uses a ring buffer with a mutex to serialize producers,
while the sole consumer, the random kthread, avoids using a mutex on the
basis that no serialization is needed since nothing else is updating the
consumer index. On platforms without total store ordering, however,
this isn't sufficient: when a producer inserts a queue entry and updates
ring.in, there is no guarantee that the consumer will see the updated
queue entry upon observing the updated producer index. That is, the
update to ring.in may be visible before the updated queue entry is
visible. As a result, we could end up mixing in zero'ed queue entries,
though this race is fairly unlikely in practice given how infrequently
the kthread runs.

The easiest way to fix this is to make the kthread acquire the mutex as
well, and hold it while processing queue entries. However, this might
result in a long hold time if there are many queue entries, and we
really want the hold times to be short, e.g., to avoid delaying
interrupt processing.

We could introduce a proper MPSC queue, but this is probably
overcomplicated for a consumer which runs at 10Hz.

Instead, define two buffers, always with one designated as the "active"
buffer. Producers queue entries in the active buffer, and the kthread
uses the mutex to atomically flip the two buffers, so it can process
entries from the inactive buffer without holding the mutex. This
requires more memory, but keeps mutex hold times short and lets us keep
the queue implementation very simple.

Reviewed by: cem
MFC after: 1 month
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D51112

(cherry picked from commit 9940c974029ba53fc00696b3fa1784725c48a9e9)