P2P ifa may require 2 routes: one is the loopback route, another is
the "prefix" route towards its destination.

Current code marks loopback routes existence with IFA_RTSELF and
"prefix" p2p routes with IFA_ROUTE.

For historic reasons, we fill in ifa_dstaddr for loopback interfaces.
To avoid installing the same route twice, we preemptively set
IFA_RTSELF when adding "prefix" route for loopback.
However, the teardown part doesn't have this hack, so we try to
remove the same route twice.

Fix this by checking if ifa_dstaddr is different from the ifa_addr
and moving this logic into a separate function.