Do control-plane route mtu tracking instead of runtime-driven one.

[melifaro]

Summary

Old behavior:
ifconfig em0 mtu 1500->9000 -> all routes traversing em0 do not change MTU.
User has to manually update all routes.
ifconfig em0 mtu 9000->1500 -> all routes traversing em0 do not change MTU.
However, if ip[6]_output finds route with rt_mtu > interface mtu, rt_mtu
gets updated.

New behavior:
ifconfig em0 mtu 1500->9000 -> all routes in all fibs gets updated with new MTU
unless RTF_FIXEDMTU flag set on them.
ifconfig em0 mtu 9000->1500 -> all routes in all fibs gets updated with new MTU
unless RTF_FIXEDMTU flag set on them.

route add ... -mtu XXX automatically sets RTF_FIXEDMTU flag.
route change .. -mtu 0 automatically removes RTF_FIXEDMTU flag.

[bz]

We have a general problem with these MTU changes and with the path MTU involved. This is only part of the solution and I would expect ip[6]_output() do adjust path MTUs and not route/interface MTUs? So I am a bit undecided if this is the right way to untangle things. I think I would rather only see MTU updates on "connected" routes (interface routes) but not on static or external ones. Now that distinction is also a bit hazy in FreeBSD.

[melifaro]

In D1125#5, @bz wrote:

We have a general problem with these MTU changes and with the path MTU involved. This is only part of the solution and I would expect ip[6]_output() do adjust path MTUs and not route/interface MTUs? So I am a bit undecided if this is the right way to untangle things. I think I would rather only see MTU updates on "connected" routes (interface routes) but not on static or external ones. Now that distinction is also a bit hazy in FreeBSD.

Well, I'm unsure if ip[6]_output should adjust path MTU (whatever it means it terms of kernel objects). Currently consumer like TCP can ask to get used @rte to check its MTU in case of EMSGSIZE and take appropriate action if needed.

Regarding routing MTU update: my primary interest is to make it calculated properly in control plane, not in data plane. We also can distinct interface-derived routes using RTF_PINNED flag (currently IPv4 only, but I'll add IPv6 support soon).

Personally I'd prefer to update all routes, not only connected ones, since it's more simple and consistent behavior.
In your case, you have to decrease MTU for ALL routes in case of ifp mtu decreased, but when you increase it back, only interface routes changes.
There are good and bad parts in both approaches, I'm not insisting on taking mine.

[melifaro]

I thought on those 2 approaches a bit more.

Different use cases I can imagine (all related to 1.5k frames and jumbo frames, ethernet medium):

"workstation" -> PC with single static/DHCP/RA-derived default route.
I'm not sure that any use case is better here.

"server" -> single static/DHCP/RA-derived default route.
User changes MTU (for the first time, for example). I'd really prefer that default route MTU stays as IS. However, we also need good docs/good "networking restart" to ease user the process of changing MTU of other routes.

"server 2" -> multiple static routes and 1500-bytes default.
In that case, updating all routes would do the right thing.

"router" (or server w/full view) -> 0 or several static routes, tons of routing-daemon derived ones.
For IGP case, routing daemon might detect MTU change (and drop adjacency, so, after adjacency is UP, routes will be added with needed MTU). However, there are cases when MTU change does not change IGP state.
For BGP case, there will be no MTU changes (which is good for eBGP but might be not so good for iBGP. However, user is capable for updating routes quickly)

Generally, better way might be "update everything but default route", however this behavior is pretty complex, so we'd better avoid it.

Other argument is that OS kernel should update routes which is installed by itself (e.g. "interface routes"). User might also expect that
static_routes (which was supplied by him to OS configs) gets update, too, but that is tricky.

So I think that @bz is right in that case and we should simply update interface routes.
One single note: I'd like to implement interface routes selection using (rt_ifp == ifp && rt_flags & (RTF_GATEWAY|RTF_HOST) == 0) approach

If there is no objections, I'd like to commit it on before the end of the week.

[melifaro]

Committed in r274611