Details

Reviewers

imp
jamie
emaste
rpaulo
jhb
allanjude
glebius

Summary

D54763 implements a workaround for ipfw KBI incompatibility introduced in D46183. It works fine for OS upgrades from 14.4 to 15.x but does not help with running an 14.4 jail with 15.0 kernel as jail initialization overwrites OS version with an older one.

This differential introduces a more confident detection through IP_FW_DUMP_SOPTCODES and IP_FW_XGET sockopts version.

Report 32-bit KBI for osreldate equal or greater than 1500034. For lower values, jailed status must be checked to make sure getosreldate() returned a real value as jail init can be instructed to override this value (see jail(8)). In case we're in a jail, use ipfw socket to detect 32-bit KBI using ophandler probes.

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

lytboris_gmail.com created this revision.Fri, Apr 24, 11:57 AM

Herald added subscribers: vegeta_tuxpowered.net, glebius, donner and 3 others. · View Herald TranscriptFri, Apr 24, 11:57 AM

lytboris_gmail.com requested review of this revision.Fri, Apr 24, 11:57 AM

The sysctl works, but I'd argue against introducing a new sysctl for this and suggest using the existing IP_FW3 sockopt interface instead.

Reasoning:

Consistency with the rest of the ipfw control plane. All ipfw configuration and introspection already goes through setsockopt(IP_FW3) / getsockopt(IP_FW3) — rules, tables, sets, skipto cache, etc. A KBI/version probe logically belongs to the same interface that the KBI itself describes.
The mechanism for version discovery already exists. IP_FW_DUMP_SOPTCODES enumerates the supported sopcodes together with their per-opcode IP_FW3_OPVER versions (see ctl_opcodes[] in ip_fw_sockopt.c). That is exactly what /sbin/ipfw needs to decide whether it is compatible with the running kernel — and it reflects reality more precisely than a single monolithic "KBI version" integer, because opcodes are versioned individually. If IP_FW_DUMP_SOPTCODES is not sufficient for the 14.4↔15.x case, it would be better to extend it (or add a dedicated IP_FW3 sopcode that returns a KBI marker) than to add a parallel discovery channel.
sysctl namespace hygiene. net.inet.ip.fw.* is user-visible and documented; a node that exists solely so that /sbin/ipfw can detect a kernel KBI quirk is implementation detail and doesn't belong in the user-facing sysctl MIB. It also has to be kept around effectively forever for ABI reasons once it ships.
Jails. The motivation in the summary is that jail init overwrites osreldate with the host-jail value. The sockopt path has no such problem and behaves identically in host and in jail, so it removes the whole class of "which knob gets overridden where" concerns — you probe the kernel you are actually talking to.

Minor nit on the patch itself: IP_FW_KBIVER=1500034 hard-codes a __FreeBSD_version-looking value. If this direction is kept, it should at least be tied to __FreeBSD_version in a comment so it is obvious when/why to bump it; otherwise it will silently rot.

Quite a trivial way to detect if we have ipfw ABI from 15.x - rely on IP_FW3_OPVER_1 for basic commands (add or get):

/*
 * ipfw_probe.c - detect new ipfw ABI by presence of XGET v=1.
 *
 * Exits with:
 *   0 - new ABI (FreeBSD 15.0+, XGET registered under version 1)
 *   1 - not new (old ABI / ipfw absent / other)
 *
 * Build:  cc ipfw_probe.c -o ipfw_probe
 * Run:    sudo ./ipfw_probe
 */

#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <netinet/in.h>
#include <netinet/ip_fw.h>

#include <err.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int
main(void)
{
        ipfw_obj_lheader hdr, *buf;
        ipfw_sopt_info *info;
        socklen_t len;
        size_t need;
        uint32_t i;
        int s, found = 0;

        s = socket(AF_INET, SOCK_RAW, IPPROTO_RAW);
        if (s < 0)
                err(2, "socket");

        memset(&hdr, 0, sizeof(hdr));
        hdr.opheader.opcode  = IP_FW_DUMP_SOPTCODES;
        hdr.opheader.version = 1;
        hdr.size             = sizeof(hdr);

        len = sizeof(hdr);
        if (getsockopt(s, IPPROTO_IP, IP_FW3, &hdr, &len) != 0 &&
            errno != ENOMEM) {
                close(s);
                return (1);
        }

        need = hdr.size;
        if (need < sizeof(hdr))
                need = sizeof(hdr);

        buf = calloc(1, need);
        if (buf == NULL)
                err(2, "calloc");
        buf->opheader.opcode  = IP_FW_DUMP_SOPTCODES;
        buf->opheader.version = 1;
        buf->size             = need;

        len = need;
        if (getsockopt(s, IPPROTO_IP, IP_FW3, buf, &len) != 0) {
                free(buf);
                close(s);
                return (1);
        }

        info = (ipfw_sopt_info *)(buf + 1);
        for (i = 0; i < buf->count; i++) {
                if (info[i].opcode == IP_FW_XGET && info[i].version == 1) {
                        found = 1;
                        break;
                }
        }

        free(buf);
        close(s);

        printf("ipfw with idx 64 bit = %d\n", found);

        return (found ? 0 : 1);
}

IMHO, Vova's plan sounds better than a sysctl.

as a proposal -

0001-ipfw-flesh-out-IP_FW3-OPVER_0-compat-layer.patch23 KBDownload

compatibility layer on top of ipfw with 64bit indexes which will provide basic functionality for 14.x and earlier clients
if you will like the idea - patch can be polished to support all scenarions

the same patch on github - https://github.com/freebsd/freebsd-src/compare/main...vgrebenschikov:freebsd-src:feature/ipfw-14-compat?expand=1

vova_fbsd.ru attached a referenced file: F153938313: 0001-ipfw-flesh-out-IP_FW3-OPVER_0-compat-layer.patch. (Show Details)Fri, Apr 24, 11:04 PM

You don't need a special sysctl for that. You can use kern.osreldate, which is generic sysctl designed exactly for such kind of problems.

But consider Vova's plan as well. Could be better.

This revision now requires changes to proceed.Sat, Apr 25, 2:52 AM

You don't need a special sysctl for that.

The fact is that I do need a separate one

You can use kern.osreldate, which is generic sysctl designed exactly for such kind of problems.

This differential was spawned by Vova's complaint that ipfw inside a 14.4 jail can not load ipfw when the kernel is 15.0. jail code (at least cbsd) overrides value of this sysctl.

lytboris_gmail.com updated this revision to Diff 176471.Sat, Apr 25, 4:13 PM

lytboris_gmail.com edited the summary of this revision. (Show Details)

Appears I was wrong. We have an instrument for jails to fake the kernel version, added in https://reviews.freebsd.org/D1948. With jails configured this way we can't use kern.osreldate to tell ipfw version.

Alas, Ian who authored D1948, is no longer with us.

Let me subscribe people who supported the feature and ask them how would they solve the problem Boris is trying to solve in this review.

Maybe we need kern.trueosreldate added?

I've tested last patch with stable/14 and in jail - it works as expected, the only proposal for stable/14 - not to try first v1 version of ABI, and instead check if we have v0 version, like:

because otherwise when used under usual 14.x kernel it throws to kernel messages
ipfw: ipfw_ctl3 invalid option 97v0

	for (i = 4; i >= 0; i--) {
		hdr = realloc(hdr, need);
		memset(hdr, 0, need);
		if (hdr == NULL)
			break;

		hdr->opheader.opcode  = IP_FW_DUMP_SOPTCODES;
		hdr->opheader.version = 0;
		hdr->size = need;

		/* Check DUMP_SOPTCODES v=0 existance */
		len = need;
		if (getsockopt(s, IPPROTO_IP, IP_FW3, hdr, &len) != 0) {
			if (errno == ENOMEM) {
				need = hdr->size;
				continue;
			}
			/* Does not exist, report 16-bit KBI */
			ret = 0;
			break;
		}
		/* fetched soptcodes successfully */
		ret = 0;
		info = (ipfw_sopt_info *)(hdr + 1);
		for (i = 0; i < hdr->count; i++) {
			if (info[i].opcode != IP_FW_XGET)
				continue;
			if (info[i].version == 0) {
				ret = 1;
				break;
			}
		}
		break;
	}

lytboris_gmail.com updated this revision to Diff 176566.Sun, Apr 26, 12:39 PM

lytboris_gmail.com edited the summary of this revision. (Show Details)

lytboris_gmail.com updated this revision to Diff 176603.Mon, Apr 27, 4:38 AM

lytboris_gmail.com edited the summary of this revision. (Show Details)

lytboris_gmail.com retitled this revision from Add sysctl with ipfw KBI version to Add ipfw 32-bit KBI via ophandler version .Mon, Apr 27, 4:49 AM

I've updated patch to minimize kernel logging errors similar to ipfw: ipfw_ctl3 invalid option 116v0 while detection is done. Unless we're not in a jail, value reported by getosreldate() is used as-is.

After applying patch - everything works as expected:

stable/14

ipfw just works, no additional messages

stable/15, main - jail with stable/14 - works, with warnings:

# cbsd jrestart f1
...
Starting jail: f1, parallel timeout=5
late_start in progress...
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Starting Network: lo0 eth0.
...
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
Flushed all rules.
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00100 allow ip from any to any via lo0
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00200 deny ip from any to 127.0.0.0/8
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00300 deny ip from 127.0.0.0/8 to any
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00400 deny ip from any to ::1
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00500 deny ip from ::1 to any
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00600 allow ipv6-icmp from :: to ff02::/16
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
00900 allow ipv6-icmp from any to any icmp6types 1
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
65000 allow ip from any to any
Firewall rules loaded.
...
Mon Apr 27 08:13:29 UTC 2026
CBSD setup: jail ipfw counters num: 99/100

next step proposal - to implement compatibility layer for earlier/historical jails

implement compatibility layer for earlier/historical jails

not in this proposal I guess.

I would mute "KBI incompatibility for ipfw is detected" for jailed environments though.

In D56616#1297786, @lytboris_gmail.com wrote:

implement compatibility layer for earlier/historical jails

not in this proposal I guess.

I would mute "KBI incompatibility for ipfw is detected" for jailed environments though.

sure, I've added - https://reviews.freebsd.org/D56660

Warning is muted for jails now

In D56616#1297887, @lytboris_gmail.com wrote:

Warning is muted for jails now

tested, yep, jail log is now clean:

# cbsd jrestart f1
...
Starting jail: f1, parallel timeout=5
...
Flushed all rules.
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
65000 allow ip from any to any
Firewall rules loaded.

Mon Apr 27 14:23:06 UTC 2026

If you find a committer (maybe @ae ?), there is no objection from my side, as the added cruft is isolated to stable/14 and won't travel with us into the future.

seems fine in concept. a few minor nits inline

sbin/ipfw/ipfw2.c
5833–5834 ↗	(On Diff #176627)
sbin/ipfw/main.c
708 ↗	(On Diff #176627)	why only `== 1`? presumably we want to warn for any instance of invoking the compat userland

lytboris_gmail.com updated this revision to Diff 176641.Mon, Apr 27, 4:26 PM

lytboris_gmail.com added inline comments.Mon, Apr 27, 4:31 PM

sbin/ipfw/main.c
708 ↗	(On Diff #176627)	Generic firewall configuration script invokes `/sbin/ipfw` for every rule so we would end up with a wall of those warnings (there's an example in "older" section of this PR). We do warn users for non-jailed cases but skip them completely for jails where user updated `osreldate` by it's own hand thus should be aware about running in compatibility mode anyway.

lytboris_gmail.com updated this revision to Diff 176646.Mon, Apr 27, 4:33 PM

lytboris_gmail.com marked an inline comment as done.Mon, Apr 27, 5:10 PM

lytboris_gmail.com retitled this revision from Add ipfw 32-bit KBI via ophandler version to Add ipfw 32-bit KBI detection via ophandler version .Tue, Apr 28, 12:43 PM

JFYI. This patch (with the one from D54763) can be applied and compiled "as-is" to stable/13 and stable/12 branches resulting a working /sbin/ipfw15 binary.

/usr/src>file /usr/obj/usr/src/amd64.amd64/sbin/ipfw/ipfw.full
/usr/obj/usr/src/amd64.amd64/sbin/ipfw/ipfw.full: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.4 (1204500), FreeBSD-style, with debug_info, not stripped

/usr/src>/usr/obj/usr/src/amd64.amd64/sbin/ipfw/ipfw.full show 2
WARNING! KBI incompatibility for ipfw is detected, trying to run /sbin/ipfw15.
execv(/sbin/ipfw15) error: No such file or directory

/usr/src>/usr/obj/usr/src/amd64.amd64/sbin/ipfw15/ipfw15.full show 2
00002 23170103  5645115489 allow ip from any to any layer2 out

stable/13 goes EOL today, a good day to merge this PR. :)

Add ipfw 32-bit KBI detection via ophandler version
Needs ReviewPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 176324

sys/netinet/ip_fw.h

sys/netpfil/ipfw/ip_fw_sockopt.c

Add ipfw 32-bit KBI detection via ophandler version Needs ReviewPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 176324

sys/netinet/ip_fw.h

sys/netpfil/ipfw/ip_fw_sockopt.c

Add ipfw 32-bit KBI detection via ophandler version
Needs ReviewPublic
Actions

Revision Contents
Changeset List