file: Add a fdclose method
Consider a program that creates a unix socket pair, transmits both
sockets from one to the other using an SCM_RIGHTS message, and then
closes both sockets without externalizing the message. unp_gc() is
supposed to handle cleanup, but it is only triggered by uipc_detach(),
which runs when a unix socket is destroyed. Because the two sockets are
internalized, their refcounts are positive, so uipc_detach() isn't
called.
As a result, a userspace program can create an unbounded amount of
garbage without triggering reclaim. Let's trigger garbage collection
whenever a unix socket is close()d. To implement this, add new a
fdclose file op and protocol op, and implement them accordingly. Since
mqueuefs has a hack to hook into the file close path, convert it to use
the new op as well.
Now, userspace can't create garbage without triggering reclamation.
Reviewed by: glebius, kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D53744
(cherry picked from commit 1390bba42caf53a00fa370f3844cd7b3725ed4ec)