zfs_mount(MS_REMOUNT): protect zfs_(un)register_callbacks calls

[avg]

Summary

We now take z_teardown_lock as a writer to ensure that there is no
I/O while the filesystem state is in a flux.
Also, zfs_suspend_fs() -> zfsvfs_teardown() call zfs_unregister_callbacks()
and zfs_resume_fs() -> zfsvfs_setup() call zfs_unregister_callbacks().
Previously there was no synchronization between those calls and the calls
in the re-mounting case. That could lead to concurrent execution
and a crash.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=180060
http://article.gmane.org/gmane.os.illumos.zfs/2833

[delphij]

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c

1256

I wonder if we should mark zfs_unregister_callbacks static, just like zfs_register_callbacks? (Or is this intentional to make them DTrace proble eligible?)

1934

I'm not very convinced that we can safely do unregister without having z_teardown_lock held here... If it's safe, maybe we should document it in the comment?

[avg]

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c

1256

It looks like this function can be made static, but I'd do it as a separate commit.

1934

If !unmounting we, in fact, do hold the lock here.
And if unmounting then no other thread can operate on the filesystem because of z_unmounted.
See line 1886 above and 1918 too.

[pho]

No problems seen with the few zfs tests I have.

[will]

LGTM.

[avg]

Committed in rS285021