MFV r275914:

As of r270383, the dbuf_compare comparator compares the dbuf
attributes in the following order:

db_level (indirect level)
db_blkid (block number)
db_state (current state)
the address of the element

Because db_state is being considered before the element's state,
changing of db_state would affect balancedness of the AVL tree,
even when the address of element compares differently. For
instance, in dbuf_create, db_state may be altered after the
node is inserted into the AVL tree and may break AVL tree
balancedness.

Instead of using db_state as a comparision critera (introduced
in r270383), consider it only when we are doing a lookup, that
is one of the two dbuf pointers contains DB_SEARCH.

Illumos issue:

5422 preserve AVL invariants in dn_dbufs

MFC after: 2 weeks