In D55203#1309731, @gallatin wrote:

I personally think NFS (or anything else) should *NOT* do this, and that it should just use mb_use_ext_pgs like sendfile does.

Resolve conflict with parent revision.

Move one line on updating rib_head from D57389 to here.

I updated my servers with this patch and got good results.
Before the patch, when I pinged routed addresses and a nexthop became unreachable, my pings got stuck.
Now it immediately shows Network is down and reacts faster when my nexthop is reachable again.
Consider I have more than 2M routes.

% ping 3fff::10:15::1
PING(56=40+8+8 bytes) 3fff::2::fbf8:3f4 --> 3fff::10:15::1
16 bytes from 3fff::10:15::1, icmp_seq=0 hlim=64 time=4.100 ms
16 bytes from 3fff::10:15::1, icmp_seq=1 hlim=64 time=4.273 ms
16 bytes from 3fff::10:15::1, icmp_seq=2 hlim=64 time=1.290 ms
16 bytes from 3fff::10:15::1, icmp_seq=3 hlim=64 time=5.433 ms
16 bytes from 3fff::10:15::1, icmp_seq=4 hlim=64 time=2.654 ms
ping: sendmsg: Network is down
ping: wrote 3fff::10:15::1 16 chars, ret=-1
ping: sendmsg: Network is down
ping: wrote 3fff::10:15::1 16 chars, ret=-1
ping: sendmsg: Network is down
ping: wrote 3fff::10:15::1 16 chars, ret=-1
16 bytes from 3fff::10:15::1, icmp_seq=8 hlim=64 time=2.928 ms
16 bytes from 3fff::10:15::1, icmp_seq=9 hlim=64 time=5.729 ms
16 bytes from 3fff::10:15::1, icmp_seq=10 hlim=64 time=2.608 ms

In D56750#1313739, @emaste wrote:

Do you have an example of the kind of backwards compat we'd provide?

netlink specs clearly want to use arp types (ARPHRD_*). (See include/uapi/linux/rtnetlink.h of linux)
However, IMO, using arp types are not appropriate here, because we actually want to get the actual interface type ifp->if_type.
therefore our current implementation (IFT_*) is correct.

Address @emaste comment.

IPv6 default route will be added using rib_default_route from route_helpers.
Ensure it sets the RT_DEFAULT_METRIC.

Fix FIB_NH_LOG format specifier.

After @zlei work on VNET. I think it's better to remove is_exclusive=true.
Tested, LGTM

Merge into D56322.

Merge D57007.
Address @markj comment.
markj asked: How do you decide whether to add a field to nhop_object vs. the other (nhop_priv)?
I reevaluate my decisions and decided to test whether if worth it to reuse spare member of nhop_object which is used for dataplane or should place it under nhop_priv?
I had two options, my initial thought was to place it inside the lookup section (NH_PRIV_END_CMP) of nhop_priv.
But since it was cache aligned, I initially decided to place metric inside the nhop_object.
But I didn't think of having to touch the cmp_priv at that time. So its cost of metric comparison by double dereferencing was not visible to me.
Now with D57007, I decided to test moving metric under nhop_priv without placing it inside the lookup section.
I decided to test that in production at NL-IX.
Here was the results:

% # When nh_metric was placed in nhop_object.
% mdo vmstat -m | grep nh
nhops   38 21248 324367067 64,128,256,512,1024,2048,4096,8192
% mdo netstat -rn4 | wc -l
1053225
% mdo netstat -rn6 | wc -l
288723
% mdo netstat -On6 | wc -l
37
% mdo netstat -on6 | wc -l
432
% # After moving it into nhop_priv (FreeBSD 16.0-CURRENT (GENERIC-NODEBUG) #3 mpath-n285993-1a086c816b5a):
% mdo vmstat -m | grep nh
nhops   38 21248 202332 64,128,256,512,1024,2048,4096,8192
% mdo netstat -rn4 | wc -l
1053251
% mdo netstat -rn6 | wc -l
288986
% mdo netstat -On6 | wc -l
40
% mdo netstat -on6 | wc -l
431

Superising to me, memory allocation is not changed.
Well, after seeing the results, I checked the size of nhop_priv again and found out the overall structure of it is not aligned to allocation boundaries.
So it is completely safe to move it under nhop_priv and it's better to do so because metric is a control-plane variable and
not belongs to nhop_object.

Merge this revision into D57011 as @markj suggested.

Merge with D57068.

Use nhop_get_metric instead of directly accessing the nh_metric.

Address @markj comment.
Remove extra macros and move metric cmp logic inside cmp_priv.
Rename cmp args.

Address @markj comments.
Initial xmin for future.
Add MPASS in nhgrp_alloc.

Address @markj comment.
Saved me, thank you.

fix format specifier to 32 bit.
@markj: Done.
Thank you

Document metric != 0 condition by replacing it with macro name.
No functional change.

Follow parent update and remove extra checking for default and wildcard metric which is now done in nhop_set_metric.
Technically no functional change.

Simplify set_metric logic by specifying 0 as RT_WILDCARD_METRIC macro and overwrite wildcard (0) metric to default metric (1).
This simple change allow users to delete all the routes without specifying metric in both rtsock and metric and also preserve POLA by not allowing users to add/change route metric to wildcard metric (0) by overwriting it to default metric.

Fix

Remove mkjail for each test and add execenv=jail to Makefile.
Address @markj comments.

Address @markj comment.
Place metric column before expire column because expire column can be hidden.

Address @markj comment.
You're right, it should, but not guaranteed.

Thank you for your review!

Address @markj comments.

LGTM

Thank you!
I'm not expert on libc, but C-wise LGTM.

Follow update in parent revision and remove metric per RTA_MULTIPATH

Enforce default metric
Update path_match_func to differenciate between nhops using metric.
Remove support for per nhop RTA_PRIORITY in RTA_MULTIPATH.

Implement support for metric via rtsock.
Add RT_DEFAULT_METRIC.