gve: Add callout to detect and handle TX timeouts
A TX timeout occurs when the driver allocates resources on a TX queue
for a packet to be sent, prompts the hardware to send the packet, but
does not receive a completion for the packet within a given timeout
period. An accumulation of TX timeouts can cause one or more queues to
run out of space and cause the entire driver to become stuck.
This commit adds a lockless timer service that runs periodically and
checks queues for timed out packets. In the event we detect a timeout,
we prompt the completion phase taskqueue to process completions. Upon
the next inspection of the queue we still detect timed out packets, if
the last "kick" occurred within a fixed cooldown window, we opt to
reset the driver, even if the prior kick successfully freed timed out
packets.
Signed-off-by: Jasper Tran O'Leary <jtranoleary@google.com>
Reviewed by: markj, ziaee
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D50385
(cherry picked from commit 3d2957336c7ddaa0a29cf60cfd458c07df1f5be9)