Embed dest sockaddr into rtentry and remove counter

[melifaro]

Summary

Currently each rtentry has dst&gateway allocated separately from another zone, bloating cache accesses.

Current 'struct rtentry' has 12 "mandatory" radix pointers in the beginning, leaving 4 usable pointers/32 bytes in the first 2 cache lines (amd64).
Fields needed for the datapath are destination sockaddr and rt_nhop.

So far it doesn't look like there is other routable addressing protocol other than IPv4/IPv6/MPLS, which uses keys longer than 20 bytes.
With that in mind, embed dst into struct rtentry, making the first 24 bytes of rtentry within 128 bytes. That is enough to make IPv6 address within first 128 bytes.

It is still pretty easy to add code for supporting separately-allocated dst, however it doesn't make a lot of sense in having such code without a use case.

As rS359823 moved the gateway to the nexthop structure, the dst embedding change moves the need for any additional allocations done by rt_setgate().

Lastly, as a part of cleanup, remove counter(9) allocation code, as the field is not used anymore.

Test Plan

divert:ipdivert_ip_input_local_success  ->  passed  [0.793s]
divert:ipdivert_ip_output_remote_success  ->  passed  [0.748s]
fibs_test:arpresolve_checks_interface_fib  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:loopback_and_network_routes_on_nondefault_fib  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:loopback_and_network_routes_on_nondefault_fib_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:same_ip_multiple_ifaces  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:same_ip_multiple_ifaces_fib0  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:same_ip_multiple_ifaces_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:slaac_on_nondefault_fib6  ->  skipped: Required configuration property 'allow_sysctl_side_effects' not defined  [0.001s]
fibs_test:subnet_route_with_multiple_fibs_on_same_subnet  ->  skipped: Required configuration property 'fibs' not defined  [0.003s]
fibs_test:subnet_route_with_multiple_fibs_on_same_subnet_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.003s]
fibs_test:udp_dontroute  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:udp_dontroute6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
forward:fwd_ip_icmp_gw_fast_success  ->  passed  [0.751s]
forward:fwd_ip_icmp_gw_slow_success  ->  passed  [0.750s]
forward:fwd_ip_icmp_iface_fast_success  ->  passed  [1.048s]
forward:fwd_ip_icmp_iface_slow_success  ->  passed  [0.840s]
ip_reass_test:ip_reass__large_fragment  ->  passed  [0.014s]
ip_reass_test:ip_reass__multiple_last_fragments  ->  passed  [0.032s]
ip_reass_test:ip_reass__zero_length_fragment  ->  passed  [0.015s]
output:output_raw_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.021s]
output:output_raw_success  ->  passed  [0.089s]
output:output_tcp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.017s]
output:output_tcp_setup_success  ->  passed  [0.138s]
output:output_udp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.019s]
output:output_udp_setup_success  ->  passed  [1.263s]
redirect:valid_redirect  ->  passed  [0.779s]
so_reuseport_lb_test:basic_ipv4  ->  passed  [0.819s]
so_reuseport_lb_test:basic_ipv6  ->  passed  [0.761s]
socket_afinet:socket_afinet  ->  passed  [0.003s]
socket_afinet:socket_afinet_bind_ok  ->  passed  [0.002s]
socket_afinet:socket_afinet_bind_zero  ->  skipped: doesn't work when mac_portacl(4) loaded (bug238781)  [0.002s]

Results file id is usr_tests_sys_netinet.20200502-183302-172181
Results saved to /home/melifaro/.kyua/store/results.usr_tests_sys_netinet.20200502-183302-172181.db

34/34 passed (0 failed)
divert:ipdivert_ip6_output_remote_success  ->  passed  [2.714s]
forward6:fwd_ip6_gu_icmp_gw_gu_fast_success  ->  passed  [2.457s]
forward6:fwd_ip6_gu_icmp_gw_gu_slow_success  ->  passed  [2.634s]
forward6:fwd_ip6_gu_icmp_gw_ll_fast_success  ->  passed  [2.331s]
forward6:fwd_ip6_gu_icmp_gw_ll_slow_success  ->  passed  [2.649s]
forward6:fwd_ip6_gu_icmp_iface_fast_success  ->  passed  [2.317s]
forward6:fwd_ip6_gu_icmp_iface_slow_success  ->  passed  [2.925s]
mld:mldraw01  ->  passed  [3.738s]
output6:output6_raw_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.025s]
output6:output6_raw_success  ->  passed  [1.893s]
output6:output6_tcp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.021s]
output6:output6_tcp_setup_success  ->  passed  [1.413s]
output6:output6_udp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.020s]
output6:output6_udp_setup_success  ->  passed  [3.400s]
redirect:valid_redirect  ->  passed  [2.728s]
scapyi386:scapyi386  ->  passed  [3.776s]

Results file id is usr_tests_sys_netinet6.20200502-183313-909256
Results saved to /root/.kyua/store/results.usr_tests_sys_netinet6.20200502-183313-909256.db

16/16 passed (0 failed)
test_rtsock_l3:rtm_add_v4_gu_ifa_ordered_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v4_gw_direct_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v4_temporal1_success  ->  passed  [0.016s]
test_rtsock_l3:rtm_add_v6_gu_gw_gu_direct_success  ->  passed  [0.007s]
test_rtsock_l3:rtm_add_v6_gu_ifa_hostroute_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v6_gu_ifa_ordered_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v6_gu_ifa_prefixroute_success  ->  passed  [0.009s]
test_rtsock_l3:rtm_add_v6_temporal1_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_del_v4_gu_ifa_prefixroute_success  ->  passed  [0.018s]
test_rtsock_l3:rtm_del_v4_prefix_nogw_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_del_v6_gu_ifa_hostroute_success  ->  passed  [0.010s]
test_rtsock_l3:rtm_del_v6_gu_ifa_prefixroute_success  ->  passed  [0.011s]
test_rtsock_l3:rtm_del_v6_gu_prefix_nogw_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_get_v4_empty_dst_failure  ->  passed  [0.003s]
test_rtsock_l3:rtm_get_v4_exact_success  ->  passed  [0.007s]
test_rtsock_l3:rtm_get_v4_hostbits_failure  ->  passed  [0.007s]
test_rtsock_l3:rtm_get_v4_lpm_success  ->  passed  [0.009s]
test_rtsock_lladdr:rtm_add_v4_gu_lle_success  ->  passed  [0.005s]
test_rtsock_lladdr:rtm_add_v6_gu_lle_success  ->  passed  [0.004s]
test_rtsock_lladdr:rtm_add_v6_ll_lle_success  ->  passed  [0.004s]
test_rtsock_lladdr:rtm_del_v4_gu_lle_success  ->  passed  [0.009s]
test_rtsock_lladdr:rtm_del_v6_gu_lle_success  ->  passed  [0.005s]
test_rtsock_lladdr:rtm_del_v6_ll_lle_success  ->  passed  [0.004s]

Results file id is usr_tests_sys_net_routing.20200502-183350-808986
Results saved to /home/melifaro/.kyua/store/results.usr_tests_sys_net_routing.20200502-183350-808986.db

23/23 passed (0 failed)

[ae]

It seems rt_pktsent is already unused in head/, thus its removal is reasonable. According to your calculations for offsets, this change can give some performance boost, and I'll try to measure it in the lab, but I'm not sure this will happen very soon. Also maybe is it worth to add some explicit alignment requirements to rtentry structure or some of its fields? We can use __aligned(CACHE_LINE_SIZE)