geom: Register classes in SI_SUB_DRIVERS as order SI_ORDER_THIRD

[stevek]

Summary

Both the devctl functionality initialization and GEOM class registration
is done during SI_SUB_DRIVERS in order SI_ORDER_SECOND. This makes them
dependent on link order as to which one gets run first. However, GEOM
class registration can cause devctl_notify() to be called. If devctl
functionality initialization has not yet been done, there will be an
attempt to lock the mutex in the devsoftc, which will be NULL. This
triggers the KASSERT() in __mtx_lock_flags.

In order to ensure devctl is initialized first, move GEOM class registration
to SI_ORDER_THIRD.

Obtained from: Juniper Networks, Inc.

[jhb]

I wonder if we could add new SI_SUB_DEVCTL_NOTIFY and SI_SUB_GEOM top level orders after SI_SUB_DRIVERS. I don't really like embedding lots of complexity among various other things within a single SYSINIT subsystem.

Doing a grep I found this nugget btw:

sys/dev/fdt/fdt_slicer.c: * Must be initialized after GEOM classes (SI_SUB_DRIVERS/SI_ORDER_SECOND),
sys/dev/fdt/fdt_slicer.c:SYSINIT(fdt_slicer, SI_SUB_DRIVERS, SI_ORDER_THIRD, fdt_slicer_init, NULL);
sys/dev/fdt/fdt_slicer.c:SYSUNINIT(fdt_slicer, SI_SUB_DRIVERS, SI_ORDER_THIRD, fdt_slicer_cleanup, NULL);
sys/dev/fdt/fdt_slicer.c:DECLARE_MODULE(fdt_slicer, fdt_slicer_mod, SI_SUB_DRIVERS, SI_ORDER_THIRD);

So you'll have to fix this one to use SI_ORDER_FOURTH, but probably this argues further for SI_SUB_GEOM.

Some more things from my grep:

sys/geom/gate/g_gate.c:DECLARE_MODULE(geom_gate, g_gate_module, SI_SUB_DRIVERS, SI_ORDER_MIDDLE);
sys/geom/raid/g_raid.c:DECLARE_MODULE(g_raid, g_raid_mod, SI_SUB_DRIVERS, SI_ORDER_THIRD);
sys/geom/raid/g_raid.h: SI_SUB_DRIVERS, SI_ORDER_SECOND);                       \
sys/geom/raid/g_raid.h: SI_SUB_DRIVERS, SI_ORDER_FIRST);                        \

Looking at all those, I think we probably don't need a separate SI_SUB_DEVCTL_NOTIFY (though I wonder why that isn't SI_ORDER_FIRST), but I do think a separate SI_SUB_GEOM for GEOM classes to use (and the fdt_slicer thing) would be sensible. It could be defined as 0x3400000 and would untangle a bit of this more cleanly I think.

[stevek]

In D44529#1020279, @jhb wrote:

Looking at all those, I think we probably don't need a separate SI_SUB_DEVCTL_NOTIFY (though I wonder why that isn't SI_ORDER_FIRST), but I do think a separate SI_SUB_GEOM for GEOM classes to use (and the fdt_slicer thing) would be sensible. It could be defined as 0x3400000 and would untangle a bit of this more cleanly I think.

I think that would be the better approach. I'll do the appropriate changes.