r343287 / D18759 introduced ifa_add_groups_to_map() which is now run by ifa_load/ifa_lookup/host_if. When loading an anchor or ruleset via pfctl that does NOT contain ifnames as hosts, host() still ends up iterating all interfaces twice, grabbing SIOCGIFGROUP ioctl twice for each. This adds an unnecessary amount of time on systems with thousands or tens of thousands of interfaces.
Prioritize the IPv4/6 check over the interface name lookup, which skips loading the iftab and iterating all interfaces when the configuration does not contain interface names.