Fix rc.firewall workstation profile for fragmented packets

[feld]

Summary

The /etc/rc.firewall workstation profile does not properly handle
fragmented packets. This causes problems for such services as
DNS and DNSSEC that may use fragmented packets.

Original Summary:
The workstation profile provided by rc.firewall breaks DNSSEC. A user
who enables the local_unbound resolver or uses another DNSSEC-aware
resolver is unable to access DNSSEC hosted services. This breaks
accessing FreeBSD.org, for example.

See Also:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216867

Test Plan

This needs to be extensibly tested to insure that it neither allows unwanted traffic, nor blocks desired traffic.

[lifanov]

Can you explain which thing you mean to fix?
I was running firewall_type="workstation" with DNSSEC validation enabled for a few years on a few machines now.

I read the forum thread and I still don't understand what the problem is.

[feld]

The discussion is here: https://forums.freebsd.org/threads/48760/

The bugzilla report is here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216867

[feld]

They're claiming some DNSSEC queries are returning fragmented as UDP and the workstation profile is dropping them. I'd like to see some packet dumps to prove it, but it feels straightforward.

[olivier]

More information about this problem created with local_unbound (that add DNSSEC support to DNS query) AND "firewall workstation" mode.

I'm working on a 12-head (r314302).

Here is my inet4 system with local_unbound configured (unbound support DNSSEC, then will receive a large, fragmented packet as response on DNSSEC enabled domain):

root@idepad:~ # grep unbound /etc/rc.conf
local_unbound_enable="YES"

IPFW is not enabled:

root@idepad:~ # sysctl net.inet.ip.fw.enable
net.inet.ip.fw.enable: 0

Name resolution is working great:

root@idepad:~ # host -a freebsd.org
Trying "freebsd.org"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26334
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;freebsd.org.                   IN      TYPE255

;; ANSWER SECTION:
freebsd.org.            2195    IN      NS      ns3.isc-sns.info.
freebsd.org.            2195    IN      NS      ns2.isc-sns.com.
freebsd.org.            2195    IN      NS      ns1.isc-sns.net.
freebsd.org.            2195    IN      RRSIG   NS 8 2 3600 20170315094057 20170301180500 19515 freebsd.org. IunQTjsGbLqcBmoQ7qiYG7WjYWQQyCBbyWWQmRFBTZNFuFdggI3KqQDl47uYyGisvurrF2fcZZaPSzQH7BSzKZnxon5jLi0IotR0P5lMJ+zF5MASl9cHo475zpWgKijbBHualxBFUeuey0eW+pP17W5b2CaeiKjDCBUE6IDbsSsFQYAi505Qy3NQ10TlFj+tJ9Ma4ebMViornco2Ih5KXPb76FBYTT7aciuHGEbuCWO5kypW1wLOqBKckG5XZY3cqpMQeBjvVOLEweQjWA9vEHHynz6i3gV0KiVnq1nldZ9qtu5Jfhc8go5Ibd2C7OeVc3IjXUa1AbOxPFAdB/dhOw==

Received 416 bytes from 127.0.0.1#53 in 0 ms

Now I'm enabling firewall in workstation mode:

sysrc firewall_enable="YES"
sysrc firewall_type="workstation"
sysrc firewall_logdeny="YES"
service ifpw start

With this standard workstation firewall, the installed rules are:

root@idepad:~ # ipfw show
00100 36  3820 allow ip from any to any via lo0
00200  0     0 deny ip from any to 127.0.0.0/8
00300  0     0 deny ip from 127.0.0.0/8 to any
00400  0     0 deny ip from any to ::1
00500  0     0 deny ip from ::1 to any
00600  0     0 allow ipv6-icmp from :: to ff02::/16
00700  0     0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800  0     0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900  0     0 allow ipv6-icmp from any to any ip6 icmp6types 1
01000  0     0 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
01100  0     0 check-state default
01200 59 11554 allow tcp from me to any established
01300  0     0 allow tcp from me to any setup keep-state default
01400 14  4274 allow udp from me to any keep-state default
01500  1    56 allow icmp from me to any keep-state default
01600  0     0 allow ipv6-icmp from me to any keep-state default
01700  0     0 allow udp from 0.0.0.0 68 to 255.255.255.255 dst-port 67 out
01800  0     0 allow udp from any 67 to me dst-port 68 in
01900  0     0 allow udp from any 67 to 255.255.255.255 dst-port 68 in
02000  0     0 allow udp from fe80::/10 to me dst-port 546 in
02100  0     0 allow icmp from any to any icmptypes 8
02200  0     0 allow ipv6-icmp from any to any ip6 icmp6types 128,129
02300  0     0 allow icmp from any to any icmptypes 3,4,11
02400  0     0 allow ipv6-icmp from any to any ip6 icmp6types 3
65000  0     0 count ip from any to any
65100  0     0 deny { tcp or udp } from any to any dst-port 135-139,445 in
65200  0     0 deny { tcp or udp } from any to any dst-port 1026,1027 in
65300  0     0 deny { tcp or udp } from any to any dst-port 1433,1434 in
65400  0     0 deny ip from any to 255.255.255.255
65500  0     0 deny ip from any to 224.0.0.0/24 in
65500  0     0 deny udp from any to any dst-port 520 in
65500  0     0 deny tcp from any 80,443 to any dst-port 1024-65535 in
65500  0     0 deny log logamount 500 ip from any to any
65535  0     0 deny ip from any to any

Now if I'm cleaning the unbound cache by restarting it and asking for a DNS query to a DNSSEC enabled domain, it will failed:

root@idepad:~ # service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 91785.
Starting local_unbound.
Waiting for nameserver to start... good
root@idepad:~ # host -a freebsd.org
Trying "freebsd.org"
;; connection timed out; no servers could be reached

And my log file will be full of "Deny UDP, frag":

root@idepad:~ # tail /var/log/security
Mar  8 00:44:04 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 13775:153@1480)
Mar  8 00:44:04 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 13785:153@1480)
Mar  8 00:44:05 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 13818:149@1480)
Mar  8 00:44:07 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 14022:153@1480)
Mar  8 00:44:07 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 14031:153@1480)
Mar  8 00:44:08 idepad kernel: ipfw: 65500 Deny UDP 192.168.100.254 192.168.100.109 in via re0 (frag 14042:153@1480)

The big DNSSEC reply, as fragmented UDP packet is rejected.
We need to reassemble packet for being allowed by ipfw.
By applying this patch, the new rule set is this one:

root@idepad:/home/olivier # ipfw show
00100  0    0 allow ip from any to any via lo0
00200  0    0 deny ip from any to 127.0.0.0/8
00300  0    0 deny ip from 127.0.0.0/8 to any
00400  0    0 deny ip from any to ::1
00500  0    0 deny ip from ::1 to any
00600  0    0 allow ipv6-icmp from :: to ff02::/16
00700  0    0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800  0    0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900  0    0 allow ipv6-icmp from any to any ip6 icmp6types 1
01000  0    0 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
01100  0    0 check-state default
01200  0    0 reass udp from any to any in
01300 30 3552 allow tcp from me to any established
01400  0    0 allow tcp from me to any setup keep-state default
01500  0    0 allow udp from me to any keep-state default
01600  0    0 allow icmp from me to any keep-state default
01700  0    0 allow ipv6-icmp from me to any keep-state default
01800  0    0 allow udp from 0.0.0.0 68 to 255.255.255.255 dst-port 67 out
01900  0    0 allow udp from any 67 to me dst-port 68 in
02000  0    0 allow udp from any 67 to 255.255.255.255 dst-port 68 in
02100  0    0 allow udp from fe80::/10 to me dst-port 546 in
02200  0    0 allow icmp from any to any icmptypes 8
02300  0    0 allow ipv6-icmp from any to any ip6 icmp6types 128,129
02400  0    0 allow icmp from any to any icmptypes 3,4,11
02500  0    0 allow ipv6-icmp from any to any ip6 icmp6types 3
65000  0    0 count ip from any to any
65100  0    0 deny { tcp or udp } from any to any dst-port 135-139,445 in
65200  0    0 deny { tcp or udp } from any to any dst-port 1026,1027 in
65300  0    0 deny { tcp or udp } from any to any dst-port 1433,1434 in
65400  0    0 deny ip from any to 255.255.255.255
65500  0    0 deny ip from any to 224.0.0.0/24 in
65500  0    0 deny udp from any to any dst-port 520 in
65500  0    0 deny tcp from any 80,443 to any dst-port 1024-65535 in
65500  0    0 deny log logamount 500 ip from any to any
65535  0    0 deny ip from any to any

With this new instruction, ipfw reassemble fragmented UDP packet (like DNSSEC reply), then it fixes this problem:

root@idepad:~ # host -a freebsd.org
Trying "freebsd.org"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51944
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;freebsd.org.                   IN      TYPE255

;; ANSWER SECTION:
freebsd.org.            3191    IN      NS      ns3.isc-sns.info.
freebsd.org.            3191    IN      NS      ns2.isc-sns.com.
freebsd.org.            3191    IN      NS      ns1.isc-sns.net.
freebsd.org.            3191    IN      RRSIG   NS 8 2 3600 20170315094057 20170301180500 19515 freebsd.org. IunQTjsGbLqcBmoQ7qiYG7WjYWQQyCBbyWWQmRFBTZNFuFdggI3KqQDl47uYyGisvurrF2fcZZaPSzQH7BSzKZnxon5jLi0IotR0P5lMJ+zF5MASl9cHo475zpWgKijbBHualxBFUeuey0eW+pP17W5b2CaeiKjDCBUE6IDbsSsFQYAi505Qy3NQ10TlFj+tJ9Ma4ebMViornco2Ih5KXPb76FBYTT7aciuHGEbuCWO5kypW1wLOqBKckG5XZY3cqpMQeBjvVOLEweQjWA9vEHHynz6i3gV0KiVnq1nldZ9qtu5Jfhc8go5Ibd2C7OeVc3IjXUa1AbOxPFAdB/dhOw==

Received 416 bytes from 127.0.0.1#53 in 207 ms

root@idepad:~ # ipfw show 1200
01200   8  1490 reass udp from any to any in

I didn't try with inet6 DNS servers.

[feld]

Thank you for taking the time to test!

I wonder if we should solve this problem by reassembling all inet4 packets by default. I've had discussions over the last few years with many people who insist that firewalls that do not reassemble inet4 packets by default should be considered broken. 😓

e.g., ${fwcmd} reass ip4 from any to any in, and place this before the check-state.

[lifanov]

I would be for reassembling all ip4 packets approach. Is there a downside to this?

[feld]

In D9920#205191, @lifanov wrote:

I would be for reassembling all ip4 packets approach. Is there a downside to this?

I am not aware of one, but I am certain we need to specifically list inet4 or ipfw breaks ipv6 traffic. I ran into that on my own systems once...

I'm also not familiar enough with the rc.firewall profiles to know if this rule should be duplicated elsewhere.

[olivier]

Good catch about only reassembling IPv4 traffic.

But about adding "inet4 reass" rules first:
ipfw man page says:

reass   Queue and reassemble IP fragments.  If the packet is not
       fragmented, counters are updated and processing continues with
       the next rule.  If the packet is the last logical fragment, the
       packet is reassembled and, if net.inet.ip.fw.one_pass is set to
       0, processing continues with the next rule.  Otherwise, the
       packet is allowed to pass and the search terminates.  If the
       packet is a fragment in the middle of a logical group of
       fragments, it is consumed and processing stops immediately.

And with these workstation rules, net.inet.ip.fw.one_pass=1.
Does this mean that ALL reassembled packet will pass ?

[freebsd_oldach.net]

Please put the reass before the check-state as fragments (except the first) don't carry protocol and port and thus cannot be dealt with by check-state anyhow. This will save a few cycles.

Furthermore I suggest to follow the storyline given in ipfw(8) and reassemble all packets (including TCP as well as IPv6):

ipfw add reass all from any to any in

[rgrimes]

I agree that the rule order is wrong here, reass should be done before a check state,
in general you want reass to occur very early, so that other rules are checking the
reassembled packet. reass should occur before any rules that list ports.

I run a couple of firewall instances that have specific rules to deal with ipv4 fragments.
I should be able to get some tcpdumps showing the fragmented traffic fairly easy.
I'll try to setup and run a test case of this patch,.
I am not so sure if we want to turn on net.inet.ip.fw.one_pass without some
additional eyes, and tests on how this might impact things.

We need additional reviews

[freebsd_oldach.net]

In D9920#307797, @rgrimes wrote:

I am not so sure if we want to turn on net.inet.ip.fw.one_pass

Indeed we should not do that. It's also not necessary. The addition of a reass line suffices to fix the issue.

BTW, the title is a bit misleading as the behavior is not directly related to DNSSEC. It's just the fragmentation of DNS packets as such - in particular with EDNS0 turned on. OARC's DNS test center provides a good description and test service: https://wwwtest.dns-oarc.net/oarc/services/replysizetest/. With the reass rule you'll get "DNS reply size limit is at least 4023 bytes" while without the rule you'll see some size around 1400.

[rgrimes]

In D9920#307806, @freebsd_oldach.net wrote:

In D9920#307797, @rgrimes wrote:

I am not so sure if we want to turn on net.inet.ip.fw.one_pass

Indeed we should not do that. It's also not necessary. The addition of a reass line suffices to fix the issue.

The reass rule has the side effect that once it assmebles a packet if net.net.ip.fw.one_pass=0 it passes
the packet without any further processing. That is not the desired behavior of a firewall.

Running a reass rule without net.net.ip.fw.one_pass=0 results in a firewall that can be
circumvented by simply fragmenting all packets.

BTW, the title is a bit misleading as the behavior is not directly related to DNSSEC. It's just the fragmentation of DNS packets as such - in particular with EDNS0 turned on. OARC's DNS test center provides a good description and test service: https://wwwtest.dns-oarc.net/oarc/services/replysizetest/. With the reass rule you'll get "DNS reply size limit is at least 4023 bytes" while without the rule you'll see some size around 1400.

Agreed, the author can update that, or I can. What should it be called?

[freebsd_oldach.net]

In D9920#307830, @rgrimes wrote:

The reass rule has the side effect that once it assmebles a packet if net.net.ip.fw.one_pass=0 it passes
the packet without any further processing. That is not the desired behavior of a firewall.

Running a reass rule without net.net.ip.fw.one_pass=0 results in a firewall that can be
circumvented by simply fragmenting all packets.

net.inet.ip.fw.one_pass=0 will not simply pass packets or fragments without any further processing through the firewall but will merely continue with the next rule in the ruleset.

Clearly one has to take the setting of net.inet.ip.fw.one_pass into account when designing the firewall ruleset.

However, the default setting is net.inet.ip.fw.one_pass=1 anyhow (at least that is what I see on 11-STABLE) and the proposed patch only adds a mandatory bit to the standard "workstation"-type firewall setup. So I guess we are pretty safe here as this is only for reasonable out-of-the-box settings.

Agreed, the author can update that, or I can. What should it be called?

"Fix rc.firewall workstation profile for fragmented packets"?

[rgrimes]

Let me try again about the net.inet.ip.fw.one_pass. This patch leaves that value alone, that value is 1 by default. The added rules shall reassemble all UDP packets, and since one_pass is set it well at that point PASS THE PACKET. This is a huge hole in the firewall in effect allowing all UDP traffic to pass inward without any port or state being checked. With the added rule that reassembles udp packets it is a MUST that net.inet.ip.fw.one_pass be set to 0 so that the additional checks later in the firewall can be checked. It is also a must that the rule be moved before the check-state.

At some point there was some concern about ipv6 and that adding the reass rule corrupted there packets, ae@ has committed a fix for that in r330792, reass now ignores ipv6 packets.

etc/rc.firewall

440

This rule needs to be moved before the check-state such that check-state can see the reassembled packet and not the packet fragments.

[freebsd_oldach.net]

In D9920#308104, @rgrimes wrote:

This patch leaves that value alone, that value is 1 by default. The added rules shall reassemble all UDP packets, and since one_pass is set it well at that point PASS THE PACKET. This is a huge hole in the firewall in effect allowing all UDP traffic to pass inward without any port or state being checked. With the added rule that reassembles udp packets it is a MUST that net.inet.ip.fw.one_pass be set to 0 so that the additional checks later in the firewall can be checked. It is also a must that the rule be moved before the check-state.

OK, got your concern. IMHO the dependency on one_pass is not particularly sensible in the reass case?

Anyhow, we can simply add ipfw disable one_pass here.