Page MenuHomeFreeBSD

if_bridge: pull up only the headers bridge_pfil() inspects
ClosedPublic

Authored by netchild on Thu, Sep 3, 6:57 AM.
Tags
None
Referenced Files
F171698691: D59332.diff
Sat, Sep 12, 7:47 PM
F171620206: D59332.id185682.diff
Sat, Sep 12, 5:51 AM
Unknown Object (File)
Fri, Sep 11, 9:53 PM
Unknown Object (File)
Fri, Sep 11, 7:29 AM
Unknown Object (File)
Fri, Sep 11, 6:28 AM
Unknown Object (File)
Thu, Sep 10, 11:11 PM
Unknown Object (File)
Thu, Sep 10, 5:48 AM
Unknown Object (File)
Wed, Sep 9, 9:57 PM

Details

Summary

bridge_pfil() pulled up min(m_pkthdr.len, max_protohdr) bytes. When the
mapped head is shorter than that and followed by an unmapped (M_EXTPG)
mbuf -- a sendfile(2) or KTLS segment from a member advertising
IFCAP_MEXTPG -- m_pullup() ran into it and dereferenced a NULL mtod(),
panicking the kernel.

Pull up the Ethernet header first, and the SNAP/LLC header only for an
802.3 frame. This is similar to pf and ip_output().

m_pullup() and m_copyup() asserted only the first mbuf; assert inside both
copy loops so the shape trips the check.

Fixes: c38abd64dbc1 ("if_epair: support IFCAP_MEXTPG")
Suggested by: markj
Assisted-by: Claude Code (Fable 5, Opus 5)

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 76744
Build 73627: arc lint + arc unit

Event Timeline

netchild held this revision as a draft.
netchild published this revision for review.Thu, Sep 3, 7:01 AM

Unit tests in D59333

After a bit more digging around. This does not seem only to be a particular problem for the bridge, this is a problem for everything pfil related...
ipfw_check_frame_mbuf() seems to have the same bug (depends on net.link.ether.ipfw=1).
pf's ethernet hook seems to be safe at first look (m_copydata()).
if_enc passes the chain without touching, safe.
dummynet: pulls 14 bytes, so probably safe.
ipfilter: no idea, maybe.

Is this the right place to fix it, or should ip_ouput() map them more early (before the PFIL hooks)?

Is this the right place to fix it, or should ip_ouput() map them more early (before the PFIL hooks)?

I take the part about ip_ouput back, it is not involved here. The failure case here is epair_tx_start_deferred → bridge_input → bridge_forward → bridge_pfil.

sys/net/if_bridge.c
3963

We have spent a fair bit of effort trying to remove these calls when possible. They are expensive and add a new failure mode. For instance, it looks like there's a bug here: we're dropping a packet but not incrementing any error counters.

Can we avoid the problem by only pulling up sizeof(struct ether_header) bytes at first, and then sizeof(struct llc) bytes later if needed?

sys/net/if_bridge.c
3963

Other areas in the bridge code seem to drop packets without increasing counters too.

I think this could be modeled here in the bridge code in a way like pf does it already in another place. But this then opens up the possibility that other places run into that same panic (see in my previous comments).

I can test easily on the system where I got the panic with pf. I have a look at coming up with something less expensive.

Narrowed per markj@: pull up ETHER_HDR_LEN first, the SNAP/LLC header only when the frame is long enough to carry one. No mb_unmapped_to_ext() call. Rebased on current main.

netchild retitled this revision from if_bridge: map unmapped mbufs before filtering to if_bridge: pull up only the headers bridge_pfil() inspects.Fri, Sep 4, 1:44 PM
netchild edited the summary of this revision. (Show Details)
netchild marked an inline comment as done.

D59389 has a similar fix for ipfw.

D59391 has patch to add error counters.

There are a couple of other places in if_bridge where we m_copyup() up to max_protohdr bytes. Do we need to fix those as well? m_copyup() doesn't handle extpg mbufs, though it wouldn't be hard to just fix that directly.

sys/net/if_bridge.c
3963

I don't think this comment is useful. It's pretty common in the network stack to only pull up the exact number of bytes you need.

There are a couple of other places in if_bridge where we m_copyup() up to max_protohdr bytes. Do we need to fix those as well? m_copyup() doesn't handle extpg mbufs, though it wouldn't be hard to just fix that directly.

Doesn't look like. My INVARIANTS kernel runs with >60 jails and KTLS+sendfile in nginx just fine. Note the added KASSERTs, they should trigger once a particular nginx jail is up.

The two with max_protohd, bridge_input() and bridge_broadcast(), cannot be reached with an unmapped chain: both run on m_dup() output.
The two in bridge_ip_checkbasic()/bridge_ip6_checkbasic() with len of sizeof(struct ip)/sizeof(struct ip6_hdr) would not trigger with the 54 bytes size I have in the unit test.

sys/net/if_bridge.c
3963

Removed locally.

sys/net/if_bridge.c
3981

Why check the full packet length? The old code doesn't do that, m_pullup() would have failed in this case. So you're changing the behaviour here, but it doesn't seem intentional.

sys/net/if_bridge.c
3981

I *think* the old code never reached that failure (min(m_pkthdr.len, max_protohdr)). For a small enough size we would get a m_pullup failed now.
A 802.1D topology-change BPDU is 14 + 3 + 4 = 21 bytes, less than ETHER_HDR_LEN + sizeof(struct llc).

I've just let Claude run some tests, not the panic trigger case, but test cases to exercise this code path (the highlight for C1 and C2 is automatic, not by me, no markup for them).
C1 is the control that proves bridge_pfil() is actually on the path (flipping pfil_onlyip alone changes the verdict),
C2 is a frame of the same 802.3 shape when it is long enough.
T is the datum: a 21-byte 802.3/LLC frame.

main without patches:

kernel  GENERIC, main-n288870-d1c07141dccf (golden image, unpatched)
C1  onlyip=1 len=21  -> DROP (expected DROP)   frames=0   "m_pullup failed"=0   OK
C2  onlyip=0 len=60  -> PASS (expected PASS)   frames=1   "m_pullup failed"=0   OK
T   onlyip=0 len=21  -> PASS (expected PASS)   frames=1   "m_pullup failed"=0   OK

This patch:

kernel  BRIDGEA, bridge-pfil-extpg-n288731-1e1f7bd9bb99 (staged 0041, guard present)
C1  onlyip=1 len=21  -> DROP (expected DROP)   frames=0   "m_pullup failed"=0   OK
C2  onlyip=0 len=60  -> PASS (expected PASS)   frames=1   "m_pullup failed"=0   OK
T   onlyip=0 len=21  -> PASS (expected PASS)   frames=1   "m_pullup failed"=0   OK

Changed patch in the sense of:

-		if ((*mp)->m_len < ETHER_HDR_LEN + sizeof(struct llc) &&
-		    (*mp)->m_pkthdr.len >= ETHER_HDR_LEN + sizeof(struct llc)) {
+		if ((*mp)->m_len < ETHER_HDR_LEN + sizeof(struct llc)) {

Result:

kernel  BRIDGEB, bridge-pfil-extpg-n288731-1e1f7bd9bb99-dirty (staged 0041, guard present)
C1  onlyip=1 len=21  -> DROP (expected DROP)   frames=0   "m_pullup failed"=1   OK
C2  onlyip=0 len=60  -> PASS (expected PASS)   frames=1   "m_pullup failed"=0   OK
T   onlyip=0 len=21  -> DROP (expected PASS)   frames=0   "m_pullup failed"=1   OK

So the code in question makes sure the behaviour does not change.

The test packet is handcrafted to test the behavior, it may not show up like that in a real system. It is supposed to test the behavior of the bridge code.

The test packet:

ff ff ff ff ff ff  02 00 00 00 00 01  00 07     dst, src, 802.3 length = 7
42 42 03                                        LLC DSAP/SSAP 0x42, control UI
00 00 00 80                                     802.1D TCN BPDU

What Claude tells about it: 14 + 3 + 4 = 21 bytes is exactly an 802.1D topology-change BPDU; an LLC XID or TEST command is shorter still. Two conditions gate reachability, both worth stating in the reply: the frame must be unpadded, which rules out anything off a real NIC (padded to 60) and rules in epair/tap/vxlan/bpf writes — what a bridge is made of; and it must not carry the 802.1D group address, which bridge_forward() drops before the filter, so the test uses broadcast.

Two by-products: on B even the C1 drop now arrives through a freed chain rather than the pfil_onlyip policy check, because the pullup runs first, and bridge_pfil()'s printf is unconditional, so the drop is also a write to the console, triggered by remote input.

And about the "unpadded" part above, the real world (partly unmapped) chain which triggers the kernel panic which triggered me into looking into this is about a 54 bytes small part of the chain, from nginx with KTLS send into the bridge:

m_pullup (n=0xfffff8054e572b00, n@entry=0xfffff80ee5695500, len=6)
        m = 0xfffff80ee5695500, space = 104, count = 60
bridge_pfil (bifp=vswitch0, ifp=j_rp_hif, dir=PFIL_IN)

kgdb printed the two mbufs. Head: m_len 54, m_pkthdr.len 925, mapped. Next: m_data NULL, m_len 871, m_flags 265 = M_EXTPG|M_RDONLY|M_EXT, with m_epg_record_type 23 and extpg_hdr 17 03 03 03 62 — a TLS 1.2 application-data record of length 0x362, i.e. KTLS. 5 + 849 + 17 = 871, and 54 + 871 = 925.

sys/net/if_bridge.c
3981

I don't know anything about llc. But remember that m_pullup will fail if you ask it to pull up more bytes than are in the mbuf chain. So it seems to be the case that a valid llc can be less than the size of the struct, in which case, m_pullup(m, 22) is naturally going to fail on a 21 byte chain, and you need to parse the size and run the pullup on the actual size. Your check seems to somehow be accidentally making the tests pass by avoiding the pullup.

sys/net/if_bridge.c
3981

The old code didn't ask in that situation too.

Old: "i = min((*mp)->m_pkthdr.len, max_protohdr)"

To me it looks like the m_pullup is not needed in the particular case. And removing that check would drop the packet instead of letting it pass.
This patch, test in previous comment:
T onlyip=0 len=21 -> PASS (expected PASS) frames=1 "m_pullup failed"=0 OK
Without the check, test in previous comment:
T onlyip=0 len=21 -> DROP (expected PASS) frames=0 "m_pullup failed"=1 OK
Vs the current main code:
T onlyip=0 len=21 -> PASS (expected PASS) frames=1 "m_pullup failed"=0 OK

I may misunderstand something here. Feel free to point to something bluntly, I'm willing to learn.

I also asked Claude some questions (and gave my and your reply),to get a better understanding of what I may be overlooking... the answer supports the need of that check:
---snip---
The code he is asking about is the second pullup the narrowed fix adds:

	if ((*mp)->m_len < ETHER_HDR_LEN + sizeof(struct llc) &&
	    (*mp)->m_pkthdr.len >= ETHER_HDR_LEN + sizeof(struct llc)) {
		*mp = m_pullup(*mp, ETHER_HDR_LEN + sizeof(struct llc));

He is right that m_pullup() would fail. The question this answers is
what "fail" costs: a crash, or a dropped frame — and whether any real
frame is in the window at all.

It drops the frame. It does not crash. m_pullup() frees the chain
and returns NULL, bridge_pfil() prints m_pullup failed and returns
-1, and all six call sites return/continue without touching *mp,
so there is no double free. Measured on an INVARIANTS + WITNESS
kernel, which would have caught one.

And the drop is a behaviour change, not a no-op. Old code passed the
frame; the guard is what keeps that true. Removing it makes a filtering
bridge start dropping frames it used to forward.
---snip---

Another answer regarding the "accidental":
---snip---
sizeof(struct llc) is 8 — if_llc.h says so itself: CTASSERT(sizeof (struct llc) == LLC_SNAPFRAMELEN), sitting next to LLC_UFRAMELEN 3, LLC_ISFRAMELEN 4, LLC_FRMRLEN 7 and the comment "Don't use sizeof(struct llc_un) for LLC header sizes". So yes, a valid LLC header can be 3 bytes and a valid frame shorter than the struct.

But there is no size to parse. LLC carries no header-length field; the only length available is the frame length, which in a chain is m_pkthdr.len — and that is what the check reads. The pullup exists solely to read llc_un.type_snap.ether_type, 8 bytes in; a frame with fewer than 8 LLC bytes cannot be SNAP, so there is nothing to make contiguous. The code doesn't look at it either: the read stays gated by m_len >= ETHER_HDR_LEN + sizeof(struct llc), which is the original test, unchanged — it was spelled with a literal 8.

It isn't accidental. The old i = min((*mp)->m_pkthdr.len, max_protohdr) is this check. The min() is exactly why the old code could never ask m_pullup() for more than the chain holds, and so never reached the free-on-short-chain path; once the requested length becomes a constant, a clamp has no expression left except a precondition. The equivalence is exact: a 21-byte frame split 14 + 7 gets pulled up to 21 by the old code, then fails m_len >= 22 and skips the SNAP read; the patch skips the pullup and fails the same test. Same verdict, one copy fewer. And that is what the T row measures — stock passes the 21-byte frame, the patch passes it, the patch without the term drops it.

If the old shape reads better, m_pullup(*mp, MIN((*mp)->m_pkthdr.len, ETHER_HDR_LEN + sizeof(struct llc))) is behaviourally identical and makes the heritage visible — it just copies a frame the following test discards anyway.
---snip---

So all in all, I do not understand how this would change behaviour or is accidental. Please be direct/blunt/... to point out what is not ok, so this crash which is currently possible in -current gets fixed the right way (you have way more experience with the network code than I do... I have just a little toe in that right now).

markj added inline comments.
sys/net/if_bridge.c
3981

I think I was missing the use of min() to pull up at most m->pkthdr.len bytes. So my original comment was wrong.

This revision is now accepted and ready to land.Wed, Sep 9, 8:33 AM
sys/net/if_bridge.c
3981

I still think the current code is broken due to the (*mp)->m_pkthdr.len >= ETHER_HDR_LEN + sizeof(struct llc)) check.
You could fail to pullup a hand-crafted packet that is ETHER_HDR_LEN + sizeof(struct llc)) -1 bytes long if you have, say, ETHER_HDR_LEN bytes in the first mbuf.

I think the min is a good idea.. eg, m_pullup(min(m->pkthdr.len bytes, sizeof (struct llc))

Limit the SNAP pullup length via min(m_pkthdr.len, ETHER_HDR_LEN + sizeof(struct llc)), per gallatin@.

This revision now requires review to proceed.Thu, Sep 10, 12:44 PM
This revision is now accepted and ready to land.Thu, Sep 10, 4:09 PM