Page MenuHomeFreeBSD

pf: change pf_route so pf only runs when packets enter and leave the stack.
ClosedPublic

Authored by kp on Apr 2 2021, 4:49 PM.

Details

Summary

before this change pf_route operated on the semantic that pf runs
when packets go over an interface, so when pf_route changed which
interface the packet was on it would run pf_test again. this change
changes (restores) the semantic that pf is only supposed to run
when packets go in or out of the network stack, even if route-to
is responsibly for short circuiting past the network stack.

just to be clear, for normal packets (ie, those not touched by
route-to/reply-to/dup-to), there isn't a difference between running
pf when packets enter or leave the stack, or having pf run when a
packet goes over an interface.

the main reason for this change is that running the same packet
through pf multiple times creates confusion for the state table.
by default, pf states are floating, meaning that packets are matched
to states regardless of which interface they're going over. if a
packet leaving on em0 is rerouted out em1, both traversals will end
up using the same state, which at best will make the accounting
look weird, or at worst fail some checks in the state and get
dropped.

another reason for this commit is is to make handling of the changes
that route-to makes consistent with other changes that are made to
packet. eg, when nat is applied to a packet, we don't run pf_test
again with the new addresses.

the main caveat with this diff is you can't have one rule that
pushes a packet out a different interface, and then have a rule on
that second interface that NATs the packet. i'm not convinced this
ever worked reliably or was used much anyway, so we don't think
it's a big concern.

discussed with many, with special thanks to bluhm@, sashan@ and
sthen@ for weathering most of that pain.
ok claudio@ sashan@ jmatthew@

Obtained from: OpenBSD
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")

Diff Detail

Repository
rG FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This revision was not accepted when it landed; it landed in state Needs Review.Apr 5 2021, 11:44 AM
This revision was automatically updated to reflect the committed changes.

Hey @kp
We've recently upgraded from FBSD 12.2 to FBSD 12.3, which contains this change. First of all: This was a breaking change for us and our rulesets didn't work anymore like before. This took us multiple hours of debugging to find the reason why. We then finally found out, that the problem is with all rules where we use route-to. Which then finally led us to this commit. After that we've found the following discussion on the mailing list: https://groups.google.com/g/bsdmailinglist/c/uMusHsnkY5s
For us it's not understandable and quite frustrating that a breaking change doesn't get announced in the relnotes or at least in the UPDATING file.

Now to the problem we have: We run pf in "block all" mode. We have an IPsec VPN (StrongSwan) that receives pakets via enc0 and pf passes them in and add's a tag (DCOUT) on the packet. Then the packet would go out on lagg0.100 but get's rerouted to lagg0.200 via a pass out rule with route-to. The route-to rule contains "flags any no state", so we do not interfere with the state table. As the last rule we have a "pass out on lagg0.200 all tagged DCOUT" which creates the correct state and passes out the paket. Due to the new behavior due to that change the last rule isn't matched anymore and the state is never setup correctly, which leds to the answer pakets being blocked by the default "block all" ruleset. Our question now is: How we can we restore the old behavior - without patching our FBSD base with reverting the change - without having to rework the whole ruleset?

Thanks and looking forward for you reply.

Hey @kp
We've recently upgraded from FBSD 12.2 to FBSD 12.3, which contains this change. First of all: This was a breaking change for us and our rulesets didn't work anymore like before. This took us multiple hours of debugging to find the reason why. We then finally found out, that the problem is with all rules where we use route-to. Which then finally led us to this commit. After that we've found the following discussion on the mailing list: https://groups.google.com/g/bsdmailinglist/c/uMusHsnkY5s
For us it's not understandable and quite frustrating that a breaking change doesn't get announced in the relnotes or at least in the UPDATING file.

Now to the problem we have: We run pf in "block all" mode. We have an IPsec VPN (StrongSwan) that receives pakets via enc0 and pf passes them in and add's a tag (DCOUT) on the packet. Then the packet would go out on lagg0.100 but get's rerouted to lagg0.200 via a pass out rule with route-to. The route-to rule contains "flags any no state", so we do not interfere with the state table. As the last rule we have a "pass out on lagg0.200 all tagged DCOUT" which creates the correct state and passes out the paket. Due to the new behavior due to that change the last rule isn't matched anymore and the state is never setup correctly, which leds to the answer pakets being blocked by the default "block all" ruleset. Our question now is: How we can we restore the old behavior - without patching our FBSD base with reverting the change - without having to rework the whole ruleset?

Thanks and looking forward for you reply.

This behaviour is not tunable, so the only option is to fix your ruleset.
It's been too long since I made this change, so the details of it have been paged out of my memory, but the archived discussion does say this is a bug fix, so I'm sorry this has affected you, but there's nothing else I can recommend other than changing your ruleset.

In D29554#793423, @kp wrote:

This behaviour is not tunable, so the only option is to fix your ruleset.
It's been too long since I made this change, so the details of it have been paged out of my memory, but the archived discussion does say this is a bug fix, so I'm sorry this has affected you, but there's nothing else I can recommend other than changing your ruleset.

Thanks for your fast reply. It might be a bug fix, but it's also a breaking change ;-). What would you recommend to simulate this behavior as we can't use "route-to" anymore (because it doesn't create the correct state)?

Thanks for your fast reply. It might be a bug fix, but it's also a breaking change ;-). What would you recommend to simulate this behavior as we can't use "route-to" anymore (because it doesn't create the correct state)?

I don't know what your current ruleset is, or what you're trying to accomplish. I also generally do not give advice on how to configure your firewall. I'm afraid I can't help you.