Ignore peer addresses in a consistent way also when checking for
new addresses during restart. If this is not done, restart doesn't
work when the local socket is IPv4 only and the peer uses
IPv4 and IPv6 addresses.

MFC after: 3 days.