In r297225 / D4306, route caching was re-enabled and added a mechanism
to cache flushed routes via tcp_ctlinput() using in_pcbnotifyall().
Unfortunately, as mentioned a few lines down in tcp_ctlinput(),
in_pcbnotifyall() is so expensive that it is almost a DOS. It holds
the tcpbinfo write lock while walking all connections. When this
happens on a heavily loaded internet facing box at Netflix with ~100K
connections, we see massive load spikes, packet drops, and throughput
loss as all CPUs spin waiting for the tcpbinfo lock.
Rather than calling in_pcbnotifyall(), let's just call tcp_notify()
directly to shoot down the routes. With this fix in place, our
IPv4 based load spikes have gone away. There are similar issues
in IPv6 which I'll address in a separate commit.