mroute: partially sanitize the file

There is rampant inconsistent formatting all around, make it mostly
style(9)-conformant.

While here:

• drop malloc casts
• rename a rw lock from mroute_mtx to mroute_lock
• replace NOTREACHED comment with __assert_unreachable

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D38652