jail: Handle jail removal in a dedicated thread

[markj]

Summary

Otherwise a deadlock is possible: the system taskqueue thread removes a
prison and calls vnet_destroy(), vnet_vlan_uninit() destroys the if_vlan
cloner, the vlan_clone_destroy() callback calls taskqueue_drain() on the
thread taskqueue.

Fix the problem by introducing a new thread for jail removals.

Ideally, the taskqueue interface would let consumers define queues
without having to map them to threads, as that'd make it possible to
avoid such deadlocks without extra threads; for now, this is the only
solution.

[jamie]

Longer term, I'd like to push some of vnet removal into the period between prison_cleanup() and the final jail destruction; interface removal is definitely part of that desire. But that's a mess of dependencies that hasn't found a solution yet, so I can see how we need this.

I don't suppose the one extra thread is a huge cost.

[markj]

In D47991#1094016, @jamie wrote:

Longer term, I'd like to push some of vnet removal into the period between prison_cleanup() and the final jail destruction; interface removal is definitely part of that desire. But that's a mess of dependencies that hasn't found a solution yet, so I can see how we need this.

I don't suppose the one extra thread is a huge cost.

It's not, but this isn't the first time we've had to resolve a deadlock this way, so it adds up over time. As I hinted in the review description, I believe the taskqueue interface needs to be simplified for low-frequency work.

[emaste]

Ideally, the taskqueue interface would let consumers define queues without having to map them to threads, as that'd make it possible to avoid such deadlocks without extra threads; for now, this is the only solution.

Is it worth extending the comment before TASKQUEUE_DEFINE_THREAD(jail_remove); with a brief note to this effect?

[markj]

In D47991#1094093, @emaste wrote:

Ideally, the taskqueue interface would let consumers define queues without having to map them to threads, as that'd make it possible to avoid such deadlocks without extra threads; for now, this is the only solution.

Is it worth extending the comment before TASKQUEUE_DEFINE_THREAD(jail_remove); with a brief note to this effect?

I think the idea is too vague to be useful or actionable, so not really suited for a comment.