Fix a panic when destroying a lagg interface.
Differential Revision: https://reviews.freebsd.org/D3883