routing: Add support for route metric

[pouria]

Summary

Apply metric to the actual list of nexthops in nhgrp.
Previously, we only supported nexthop weight for ECMP routes.
The upper 8 bit of weight was reserved for metric or administrative
distance purposes.
Now we filter any nexthops with lower metric in the actual nexthop
group.

[pouria]

Here is how it works:

[root@ftsr1] [~] # route -n6 add -net 3fff::/64 -gateway 2a01:e140:10:10::6 -weight 33554435
change net 3fff::/64: gateway 2a01:e140:10:10::6 fib 0
[root@ftsr1] [~] # route -n6 add -net 3fff::1/64 -gateway 2a01:e140:10:10::1 -weight 2
add net 3fff::1/64: gateway 2a01:e140:10:10::1 fib 0
[root@ftsr1] [~] # route -n6 get 3fff::/64
   route to: 3fff::
destination: 3fff::
       mask: ffff:ffff:ffff:ffff::
    gateway: 2a01:e140:10:10::6
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu       metric    weight    expire
       0         0         0         0      1500         2         3         0
[root@ftsr1] [~] # route -on6 get 3fff::/64
   route to: 3fff::
destination: 3fff::
       mask: ffff:ffff:ffff:ffff::
        fib: 0
      flags: <UP,GATEWAY,DONE,STATIC>
      nhops: 2
        via gw 2a01:e140:10:10::1 iface vtnet0 metric 0 weight 2 mtu 1500 table inet6.0
        via gw 2a01:e140:10:10::6 iface vtnet0 metric 2 weight 3 mtu 1500
[root@ftsr1] [~] # netstat -rn6W | grep 3fff
3fff::/64                         2a01:e140:10:10::1            UGS         0   1500           vtnet0
3fff::/64                         2a01:e140:10:10::6            UGS         0   1500           vtnet0
[root@ftsr1] [~] # netstat -On6W
Nexthop groups data

Internet6:
GrpIdx  NhIdx     Weight   Slots                                 Gateway     Netif  Refcnt
11        ------- ------- ------- --------------------------------------- ---------       2
               9       2       0                      2a01:e140:10:10::1    vtnet0
              1033554435      64                      2a01:e140:10:10::6    vtnet0

Of course, we should add a separate -metric argument to route(8) later.

sys/net/route.h

107–108

Here is another hint for me to understand what was the original @melifaro design.
See below: get_info_weight

sys/net/route/nhgrp_ctl.c

147–148

Changes here only affect the actual nhgrp size in alloc_nhgrp().
So, basically, we're reducing the size required for mpath if metric sets.

252

This is not necessary.
I can revert it.

sys/net/route/route_ctl.c

199

Why not use a separate value for metric?
Here is the reason.
Both approaches had its own side effects.
I decided to go with original @melifaro design.

[pouria]

Also this revision will NOT change expected routing behavior (POLA) since we never allow users to set weight more than 24-bit.
Now we don't, but we use it for metric.
I intentionally used the term metric instead of administrative distance. You know the difference.

before this patch, adding nexthop never results in change to the main or first nhop of the nexthop group.
Before this patch:

[pouria@cornelia] [/usr/src] [mpath_user|✔] % mdo route -n6 add -host 3fff::1 -gateway ::4 -weight 16777218
add host 3fff::1: gateway ::4
[pouria@cornelia] [/usr/src] [mpath_user|✔] % netstat -On6W
Nexthop groups data

Internet6:
GrpIdx  NhIdx     Weight   Slots                                 Gateway     Netif  Refcnt
14        ------- ------- ------- --------------------------------------- ---------       2
               7       1       0                                     ::1       lo0
               8       2       0                                     ::2       lo0
              11 16777218      32                                     ::4       lo0
              12 16777216      32                                     ::3       lo0
[pouria@cornelia] [/usr/src] [mpath_user|✔] % echo '2^24' | bc
16777216
[pouria@cornelia] [/usr/src] [mpath_user|✔] % echo '(2^24)*2' | bc
33554432
[pouria@cornelia] [/usr/src] [mpath_user|✔] % mdo route -n6 add -host 3fff::1 -gateway ::5 -weight 33554434
add host 3fff::1: gateway ::5
[pouria@cornelia] [/usr/src] [mpath_user|✔] % route -n6 get 3fff::1
   route to: 3fff::1
destination: 3fff::1
    gateway: ::4
        fib: 0
  interface: lo0
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         0         0
[pouria@cornelia] [/usr/src] [mpath_user|✔] % netstat -On6W
Nexthop groups data

Internet6:
GrpIdx  NhIdx     Weight   Slots                                 Gateway     Netif  Refcnt
15        ------- ------- ------- --------------------------------------- ---------       2
               7       1       0                                     ::1       lo0
               8       2       0                                     ::2       lo0
              11 16777218      16                                     ::4       lo0
              12 16777216      15                                     ::3       lo0
              13 33554434      33                                     ::5       lo0

You may wonder how I set the weight to more than 24-bit before this patch?
The answer is we didn't validate maximum weight value from netlink.
However, we didn't document any of the weight system. So we're safe from breaking the userland.

After this patch:

[root@ftsr1] [~] # route -n6 add -net 3fff::/64 -gateway ::4 -weight 16777218
add net 3fff::/64: gateway ::4 fib 0
[root@ftsr1] [~] # netstat -On6W
Nexthop groups data

Internet6:
GrpIdx  NhIdx     Weight   Slots                                 Gateway     Netif  Refcnt
14        ------- ------- ------- --------------------------------------- ---------       2
               9       2       0                                     ::1       lo0
              10       2       0                                     ::2       lo0
              1116777218      32                                     ::4       lo0
              1216777216      32                                     ::3       lo0
[root@ftsr1] [~] # route -n6 add -net 3fff::/64 -gateway ::5 -weight 33554433
add net 3fff::/64: gateway ::5 fib 0
[root@ftsr1] [~] # netstat -On6W
Nexthop groups data

Internet6:
GrpIdx  NhIdx     Weight   Slots                                 Gateway     Netif  Refcnt
15        ------- ------- ------- --------------------------------------- ---------       2
               9       2       0                                     ::1       lo0
              10       2       0                                     ::2       lo0
              1116777218       0                                     ::4       lo0
              1216777216       0                                     ::3       lo0
              1333554433      64                                     ::5       lo0

Compare the nexthop slots of the nexthop group, before and after this patch.

[pouria]

Refactor to use union instead.
This is much more cleaner.

[pouria]

Update structure to make it compile separately.
+ fix minor issues.

[pouria]

sys/net/route/route_helpers.c

297

With new structure, this initialization must be changed.
This is NOT for styling.