Page MenuHomeFreeBSD

Embed dest sockaddr into rtentry and remove counter
ClosedPublic

Authored by melifaro on May 2 2020, 6:13 PM.

Details

Summary

Currently each rtentry has dst&gateway allocated separately from another zone, bloating cache accesses.

Current 'struct rtentry' has 12 "mandatory" radix pointers in the beginning, leaving 4 usable pointers/32 bytes in the first 2 cache lines (amd64).
Fields needed for the datapath are destination sockaddr and rt_nhop.

So far it doesn't look like there is other routable addressing protocol other than IPv4/IPv6/MPLS, which uses keys longer than 20 bytes.
With that in mind, embed dst into struct rtentry, making the first 24 bytes of rtentry within 128 bytes. That is enough to make IPv6 address within first 128 bytes.

It is still pretty easy to add code for supporting separately-allocated dst, however it doesn't make a lot of sense in having such code without a use case.

As rS359823 moved the gateway to the nexthop structure, the dst embedding change moves the need for any additional allocations done by rt_setgate().

Lastly, as a part of cleanup, remove counter(9) allocation code, as the field is not used anymore.

Test Plan
divert:ipdivert_ip_input_local_success  ->  passed  [0.793s]
divert:ipdivert_ip_output_remote_success  ->  passed  [0.748s]
fibs_test:arpresolve_checks_interface_fib  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:loopback_and_network_routes_on_nondefault_fib  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:loopback_and_network_routes_on_nondefault_fib_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:same_ip_multiple_ifaces  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:same_ip_multiple_ifaces_fib0  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:same_ip_multiple_ifaces_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.001s]
fibs_test:slaac_on_nondefault_fib6  ->  skipped: Required configuration property 'allow_sysctl_side_effects' not defined  [0.001s]
fibs_test:subnet_route_with_multiple_fibs_on_same_subnet  ->  skipped: Required configuration property 'fibs' not defined  [0.003s]
fibs_test:subnet_route_with_multiple_fibs_on_same_subnet_inet6  ->  skipped: Required configuration property 'fibs' not defined  [0.003s]
fibs_test:udp_dontroute  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
fibs_test:udp_dontroute6  ->  skipped: Required configuration property 'fibs' not defined  [0.002s]
forward:fwd_ip_icmp_gw_fast_success  ->  passed  [0.751s]
forward:fwd_ip_icmp_gw_slow_success  ->  passed  [0.750s]
forward:fwd_ip_icmp_iface_fast_success  ->  passed  [1.048s]
forward:fwd_ip_icmp_iface_slow_success  ->  passed  [0.840s]
ip_reass_test:ip_reass__large_fragment  ->  passed  [0.014s]
ip_reass_test:ip_reass__multiple_last_fragments  ->  passed  [0.032s]
ip_reass_test:ip_reass__zero_length_fragment  ->  passed  [0.015s]
output:output_raw_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.021s]
output:output_raw_success  ->  passed  [0.089s]
output:output_tcp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.017s]
output:output_tcp_setup_success  ->  passed  [0.138s]
output:output_udp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.019s]
output:output_udp_setup_success  ->  passed  [1.263s]
redirect:valid_redirect  ->  passed  [0.779s]
so_reuseport_lb_test:basic_ipv4  ->  passed  [0.819s]
so_reuseport_lb_test:basic_ipv6  ->  passed  [0.761s]
socket_afinet:socket_afinet  ->  passed  [0.003s]
socket_afinet:socket_afinet_bind_ok  ->  passed  [0.002s]
socket_afinet:socket_afinet_bind_zero  ->  skipped: doesn't work when mac_portacl(4) loaded (bug238781)  [0.002s]

Results file id is usr_tests_sys_netinet.20200502-183302-172181
Results saved to /home/melifaro/.kyua/store/results.usr_tests_sys_netinet.20200502-183302-172181.db

34/34 passed (0 failed)
divert:ipdivert_ip6_output_remote_success  ->  passed  [2.714s]
forward6:fwd_ip6_gu_icmp_gw_gu_fast_success  ->  passed  [2.457s]
forward6:fwd_ip6_gu_icmp_gw_gu_slow_success  ->  passed  [2.634s]
forward6:fwd_ip6_gu_icmp_gw_ll_fast_success  ->  passed  [2.331s]
forward6:fwd_ip6_gu_icmp_gw_ll_slow_success  ->  passed  [2.649s]
forward6:fwd_ip6_gu_icmp_iface_fast_success  ->  passed  [2.317s]
forward6:fwd_ip6_gu_icmp_iface_slow_success  ->  passed  [2.925s]
mld:mldraw01  ->  passed  [3.738s]
output6:output6_raw_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.025s]
output6:output6_raw_success  ->  passed  [1.893s]
output6:output6_tcp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.021s]
output6:output6_tcp_setup_success  ->  passed  [1.413s]
output6:output6_udp_flowid_mpath_success  ->  skipped: This test requires ROUTE_MPATH enabled  [0.020s]
output6:output6_udp_setup_success  ->  passed  [3.400s]
redirect:valid_redirect  ->  passed  [2.728s]
scapyi386:scapyi386  ->  passed  [3.776s]

Results file id is usr_tests_sys_netinet6.20200502-183313-909256
Results saved to /root/.kyua/store/results.usr_tests_sys_netinet6.20200502-183313-909256.db

16/16 passed (0 failed)
test_rtsock_l3:rtm_add_v4_gu_ifa_ordered_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v4_gw_direct_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v4_temporal1_success  ->  passed  [0.016s]
test_rtsock_l3:rtm_add_v6_gu_gw_gu_direct_success  ->  passed  [0.007s]
test_rtsock_l3:rtm_add_v6_gu_ifa_hostroute_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v6_gu_ifa_ordered_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_add_v6_gu_ifa_prefixroute_success  ->  passed  [0.009s]
test_rtsock_l3:rtm_add_v6_temporal1_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_del_v4_gu_ifa_prefixroute_success  ->  passed  [0.018s]
test_rtsock_l3:rtm_del_v4_prefix_nogw_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_del_v6_gu_ifa_hostroute_success  ->  passed  [0.010s]
test_rtsock_l3:rtm_del_v6_gu_ifa_prefixroute_success  ->  passed  [0.011s]
test_rtsock_l3:rtm_del_v6_gu_prefix_nogw_success  ->  passed  [0.006s]
test_rtsock_l3:rtm_get_v4_empty_dst_failure  ->  passed  [0.003s]
test_rtsock_l3:rtm_get_v4_exact_success  ->  passed  [0.007s]
test_rtsock_l3:rtm_get_v4_hostbits_failure  ->  passed  [0.007s]
test_rtsock_l3:rtm_get_v4_lpm_success  ->  passed  [0.009s]
test_rtsock_lladdr:rtm_add_v4_gu_lle_success  ->  passed  [0.005s]
test_rtsock_lladdr:rtm_add_v6_gu_lle_success  ->  passed  [0.004s]
test_rtsock_lladdr:rtm_add_v6_ll_lle_success  ->  passed  [0.004s]
test_rtsock_lladdr:rtm_del_v4_gu_lle_success  ->  passed  [0.009s]
test_rtsock_lladdr:rtm_del_v6_gu_lle_success  ->  passed  [0.005s]
test_rtsock_lladdr:rtm_del_v6_ll_lle_success  ->  passed  [0.004s]

Results file id is usr_tests_sys_net_routing.20200502-183350-808986
Results saved to /home/melifaro/.kyua/store/results.usr_tests_sys_net_routing.20200502-183350-808986.db

23/23 passed (0 failed)

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

melifaro retitled this revision from Optimize rtenty fields accesses and embed dst. to Embed dest sockaddr into rtentry and remove counter.May 2 2020, 6:34 PM
melifaro edited the summary of this revision. (Show Details)
melifaro edited the test plan for this revision. (Show Details)
melifaro added a reviewer: network.

It seems rt_pktsent is already unused in head/, thus its removal is reasonable. According to your calculations for offsets, this change can give some performance boost, and I'll try to measure it in the lab, but I'm not sure this will happen very soon. Also maybe is it worth to add some explicit alignment requirements to rtentry structure or some of its fields? We can use __aligned(CACHE_LINE_SIZE)

This revision was not accepted when it landed; it landed in state Needs Review.May 8 2020, 9:06 PM
This revision was automatically updated to reflect the committed changes.