nda: Don't sleep with non-sleepable lock held

[cperciva]

Summary

We reach ndaasync with the CAM device lock held, so we must pass
M_NOWAIT to disk_* rather than M_WAITOK.

Fixes: 628d7a3270b6 ("nda: AC_GETDEV_CHANGED calls media chanaged for sectorsize change")
MFC after: 1 week
Sponsored by: Amazon

[cperciva]

@imp In 824530a5c4fd59427cda0eae0a4ac85212814958 you wrote nda: Assume all cases in ndaasync can sleep and changed a different M_NOWAIT to M_WAITOK, but WITNESS complained about this code path:

Jul 14 08:32:08 freebsd kernel: uma_zalloc_debug: zone "malloc-256" with the following non-sleepable locks held:
Jul 14 08:32:08 freebsd kernel: exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff801ae8e24d0) locked @ /usr/src/sys/cam/cam_xpt.c:2238
Jul 14 08:32:08 freebsd kernel: stack backtrace:
Jul 14 08:32:08 freebsd kernel: #0 0xffffffff80c8b62c at witness_debugger+0x6c
Jul 14 08:32:08 freebsd kernel: #1 0xffffffff80c8cf2c at witness_warn+0x4bc
Jul 14 08:32:08 freebsd kernel: #2 0xffffffff810002bf at uma_zalloc_debug+0x3f
Jul 14 08:32:08 freebsd kernel: #3 0xffffffff80fffe07 at uma_zalloc_arg+0x27
Jul 14 08:32:08 freebsd kernel: #4 0xffffffff80bdcd1d at malloc+0x7d
Jul 14 08:32:08 freebsd kernel: #5 0xffffffff80b31e8c at g_post_event+0x7c
Jul 14 08:32:08 freebsd kernel: #6 0xffffffff803b43d2 at ndaasync+0x192
Jul 14 08:32:08 freebsd kernel: #7 0xffffffff803ab4ba at xpt_async_process_dev+0x1ba
Jul 14 08:32:08 freebsd kernel: #8 0xffffffff803aa30c at xptdevicetraverse+0x9c
Jul 14 08:32:08 freebsd kernel: #9 0xffffffff803aa0b4 at xpttargettraverse+0x74
Jul 14 08:32:08 freebsd kernel: #10 0xffffffff803a6b1a at xpt_async_process+0x26a
Jul 14 08:32:08 freebsd kernel: #11 0xffffffff803a73a0 at xpt_done_process+0x3e0
Jul 14 08:32:08 freebsd kernel: #12 0xffffffff803a9775 at xpt_async_td+0xe5
Jul 14 08:32:08 freebsd kernel: #13 0xffffffff80bba112 at fork_exit+0x82
Jul 14 08:32:08 freebsd kernel: #14 0xffffffff8116961e at fork_trampoline+0xe
Jul 14 08:32:08 freebsd kernel: GEOM_PART: nda0 was automatically resized.
Jul 14 08:32:08 freebsd kernel:   Use `gpart commit nda0` to save changes or `gpart undo nda0` to revert them.

I'm not 100% certain if this same backtrace would apply to the AC_ADVINFO_CHANGED code path but perhaps that change needs to be reverted too?

[imp]

So this is OK. A better thing would be to just drop the CAM lock. That should be safe. However, it's going to be super rare we can't allocate here. Can you commit this as is, but file a bug and assign it to me. The bug should be 'investigate dropping CAM lock for AC_GETDEV_CHANGED in nda' and I'll not forget about it that way.

[cperciva]

In D58230#1341164, @imp wrote:

So this is OK. A better thing would be to just drop the CAM lock. That should be safe. However, it's going to be super rare we can't allocate here. Can you commit this as is, but file a bug and assign it to me. The bug should be 'investigate dropping CAM lock for AC_GETDEV_CHANGED in nda' and I'll not forget about it that way.

Thanks, I've opened https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297075 and assigned it to you.