lockf: slightly depessimize

1. check if P_ADVLOCK is already set and if so, don't lock to set it

(stolen from DragonFly)

2. when trying for fast path unlock, check that we are doing unlock

first instead of taking the interlock for no reason (e.g. if we want
to *lock*). whilere make it more likely that falling fast path will
not take the interlock either by checking for state

Note the code is severely pessimized both single- and multithreaded.