casper(3):
If nvlout is an error state before we attempt to send it, rebuild with just the final error to be passed back. This can happen for various reasons, but most of them seem to be either EMFILE/ENFILE/ENOMEM issues that would naturally clear up by creating a more minimal nvlist with a single nvpair. Regardless, we must try- libnv will not send out an nvlist that's in an error state.
cap_fileargs(3):
Attempt to handle EMFILE/ENFILE style errors in fileargs_add_cache. In one error case, the nvlist_add_nvlist(nvlout, fname, new) was setting EMFILE on nvlout and causing the send to later fail.
libnv(3):
recvmsg can kick back EMSGSIZE if the SCM_RIGHTS transfer can't happen due to EMFILE/ENFILE. Drain the recv buf if we've hit EMSGSIZE so that subsequent transfers can work as expected, rather than the current domino effect.
I'm not sure any of it's the most correct; the libnv(3) change could/should probably only drain enough bytes to hold the cmsg that can't happen (nfds * CMSG_SPACE(sizeof(int)) + sizeof(struct msghdr).