ifconfig: also fix removing IPv6 addresses without netlink

Reported by: bdrewery
MFC after: 1 week
See also: da50f49977cc4e6aae55cb2379313599249a8dd2
Sponsored by: Rubicon Communications, LLC ("Netgate")

(cherry picked from commit c8fb5a4d032b28dd97b9fa4a73f85a84d8d35fdd)