Consider this scenario:
- kern.corefile=/var/coredumps/%N.%U.%I.core
- multiple processes with the same name crash at the same time
It's possible that one process selects existing file N as oldvp while it
keeps looking for an unused file number. Another process scans through
files and stumbles upon N. That process would be blocked on the vnode lock
while holding the directory vnode exclusively locked. The first process
would, thus, get blocked on the director's vnode lock.
More generally, holding a file's vnode lock (oldvp) while trying to lock its
directory (for the next lookup) is a violation of the vnode locking order.
I have observed this deadlock in the wild.
So, the change to keep oldvp "opened" but unlocked and to lock it again only
if it's to be returned as the result.