netlink: answer a request whose reply ran out of memory

[rcm]

Summary

A writer that fails to allocate a buffer refuses every later message,
the ack and NLMSG_DONE included. nlmsg_end_dump() ignored that,
reported success and suppressed the ack. Either way the request got no
answer and its sender waited for ever. The requests of a whole send(2)
were lost the same way when the first buffer of the writer could not be
allocated.

Neither that buffer nor the ack is allocated with locks held, so let
both wait for memory. Send the ack with ENOMEM for every reply that
ran out of memory: that also ends a dump that nlmsg_end_dump() could
not.

Replies to Linux processes can still be lost when their translation
runs out of memory.

Fixes: 7e5bf68495cc ("netlink: add netlink support")
Fixes: f4d3aa749084 ("netlink: suppress sending NLMSG_ERROR if NLMSG_DONE is already sent")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

[rcm]

The first version made room for NLMSG_DONE by overwriting completed messages in the writer's buffer, which could discard replies to earlier requests from the same send() and still left non-dump requests without an ack. This version drops that: nlmsg_ack() now waits for memory and reports ENOMEM for any reply that ran out of it, so a failed dump ends with NLMSG_ERROR after its complete messages and every request gets an answer.