Missed part of r356162.

If we postpone consumer destruction till close, then the close calls should
not be ignored. Delay geom withering till the last close too.

MFC after: 2 weeks
X-MFC-with: r356162
Sponsored by: iXsystems, Inc.