FYI: @bms, I redesigned the current fib_algo to support neighbor families other than extra 4o6. Subhash Murmu (MPLS GSoC student) can increase FD_MAX_NH_AF and reuse it for its own dataplane.

I think I should explain what happens here, I'll start from relationships between routing modules:

In the networking industry there is a clear separation between RIB and FIB, at least as far as vendors are concerned.
In the FreeBSD context, the RIB is the control-plane of the routing table (route_ctl.c, route.c, ...).

For routing lookups, though, we can't simply call rib_lookup_*() or rnh->rnh_lookup(), because that is too slow for the data plane.
We need a summarized table for that (e.g. Cisco has CEF for this).
This is where the FIB helps: we can use a radix[4|6] tree and hook into control-plane changes (route_subscription, from fib_algo to route_ctl.c) to build a simple, fast table used only for data-plane lookups.

But we may want to trade speed against memory depending on the number of routes, so we need to be able to change the lookup algorithm.
That is the fib_algo framework: an abstraction layer for FIB lookup algorithms.
Modules like dxr (sys/netinet/in_fib_dxr.c) can then ignore nhop referencing and control-plane (RIB) subscriptions and focus on implementing their own lookup algorithms.

Now that the relationship is clear, let's look at the problem.

The fib_algo framework is responsible for managing nhop references.
To keep it simple and fast, it allocates an initial array of 16 nhops and doubles it every time that limit is reached.
The array is fast because it is indexed directly by the nhop id (nhop_var.h: struct nhop_priv->nh_idx), on the assumption that the id is unique.
That assumption holds, because both nhop and nhgrp get their indexes from their nh_control, and nh_control guarantees uniqueness. nh_control is per family, just like the fib_algo instances.

When a new route is added, the nhop allocation takes its family from rib_family (nhop_ctl.c:710) and stores it in nh_priv->nh_upper_family.
However, the hash table and index management for a nexthop use nh_neigh_family, which can differ from the upper family (nhop.c:338).

That is what breaks the assumption. As today IPv4 over IPv6 nexthop.

If the bug is triggered in a production environment, what is the likely outcome?

Technically it's an index corruption.

• sys/net/route/fib_algo.c:1110 (free based on idx, invalid refcount, you know the rest)
• sys/net/route/fib_algo.c:1754 (returns a completely wrong nexthop from fib_lookup(), which usually results in dropped packets)
• sys/net/route/fib_algo.c:1784 (refcounts the wrong nexthop, you know the rest)
• sys/net/route/fib_algo.c:1845 ( this is where the KASSERT fires, unreferencing the wrong nexthop)

Hope it helps the reviewers.