Move swi_remove() call before acquiring the tty lock. swi_remove() calls intr_event_remove_handler() which may sleep via msleep(), causing a lock order violation when called with the tty mutex held.
The software interrupt handler removal operates on the interrupt event structure independently and does not require the tty lock. This matches the pattern used in other drivers such as tcp_hpts.c where swi_remove() is called without holding other locks.