Page MenuHomeFreeBSD

Implement ECN++ (draft-generalized-ecn)
Needs ReviewPublic

Authored by rscheff on Jan 17 2020, 11:28 AM.
Referenced Files
Unknown Object (File)
Sun, Dec 10, 1:32 AM
Unknown Object (File)
Thu, Nov 30, 1:13 AM
Unknown Object (File)
Wed, Nov 29, 3:21 AM
Unknown Object (File)
Mon, Nov 27, 1:14 PM
Unknown Object (File)
Fri, Nov 24, 6:40 PM
Unknown Object (File)
Thu, Nov 23, 10:20 AM
Unknown Object (File)
Thu, Nov 23, 12:29 AM
Unknown Object (File)
Wed, Nov 22, 8:08 AM



RFC3168 only allows regular data segments to be sent as
ECN-capable transport (ECT). Control (RST, FIN), pure ACK
and retransmissions are explicitly not to be sent with ECT.

However, it has been shown over the years, that this
approach stifles the usefulness of ECN, in particular when
deploying DCTCP in controlled environments.

Current work in the IETF (draft-generalized-ecn, also known
as ECN++) aims to allow all control and retransmission
packets to be marked ECT. In the case of retransmissions,
the only standardized recourse when one of those is dropped,
is to wait for the RTO timer to expire - including the
penalty imposed by having to suffer an RTO.

Furthermore, allowing control and pure ACKs to convey
additional congestion signals on the return path enables
features like AckCC (RFC5690).

Also, adding code to make non-session related control packets
(eg. RST to non-listening ports) appear idential with those in
response to recently used sessions, to prevent sidechannel
information leakage.

Test Plan

Attaching packetdrill scripts for full coverage

Diff Detail

rG FreeBSD src repository
Lint Passed
No Test Coverage
Build Status
Buildable 48244
Build 45130: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Check what Linux / MacOS are doing in this space, may be beneficial to keep all this to a single sysctl instead.

Discussed with the people doing the TCP Prague / DCTCP implementation on Linux; the code there doesn't use an expicit, separate sysctl yet - all experimentation is done with hardcoded enabling of ECN and IP header markings, when their CC variant is active.

Also, RFC5562 has some additional handshake requirements (apparently added after the initial simulation, and without vetting against real-world issues observed in the internet, e.g. overriding the legacy and historic IP TOS byte with bit patterns, which effectively keep marking every packet as CE...

Due to these shortcomings of RFC5562 (known incompabilitiy to a rare, but observable misconfiguration of devices on the internet, and no other expirience in deployed stacks, will remove the "ECN+" (RFC5562) signaling part.

The sysctl will only enable full signaling according to draft-ietf-tcpm-generalized-ecn, with ecn.enable still being the "main switch".

After this diff, will add a patch to automatically enable ecn and ecnplusplus with the dctcp cc loadable module, to get these settings enabled in a VNET automatically, reducing chances that DCTCP is run without ECN, or with poor session setup performance under load (without ECN++).

  • rebase
  • removing incomplete ECN+ code
  • adding sysctl dependencies
  • deal with non-socket related control packets
rscheff retitled this revision from Implement ECN+ (RFC5562) and ECN++ feature (draft-generalized-ecn) to Implement ECN++ (draft-generalized-ecn) .Feb 8 2021, 3:05 PM
rscheff edited the summary of this revision. (Show Details)
rscheff retitled this revision from Implement ECN++ (draft-generalized-ecn) to Implement ECN++ (draft-generalized-ecn).
  • updated timestamp in man file
  • rebase
  • streamline fastpath check
  • update man date
  • have to explicitly check for ECN SYN
  • add TF2_ECN_PLUSPLUS flag to reduce cacheline churn in fastpath when checking V_ecn_generalized
  • rebase
  • bump tcp.4 datestamp added inline comments.

Typo and move full stop to a more natural place.

Current work in the IETF (draft-generalized-ecn, also known as ECN++) does not return a document.
The IETF TSV WG is where I would expect any such work to be done, but does not appear to show
any documents that might be given the moniker "ECN++". Could you provide
a better reference or link for the draft specification that is being implemented?

Ah, a TCP-specific draft, not generic to all ECN usage. So I was looking in the wrong place :)
Thanks for the pointer.

LGTM now.

By which I mean, the language in the manual page change does. Can't say more.

Typo in man page.



OK for the manpages part of the change.

  • rebase
  • simplify the fastpath check if ECT should be set.
  • streamline fastpath check
  • update man date
  • have to explicitly check for ECN SYN
  • add TF2_ECN_PLUSPLUS flag to reduce cacheline churn in fastpath when checking V_ecn_generalized
  • bump tcp.4 datestamp
  • fix man page typos
  • fix typo in man page
  • update to use the common tcp_ecn.c sourcefile
  • fix types

Can you explain what you mean by that? I'm arguably not the indended audience, but if I don't understand it, there's a chance some of the intended audience and or some interested readers won't either.


When a host receives a TCP packet to a port which is not listening, or no connection exists, or the header information is in some other way not acceptable, the host may respond with a RST (reset) packet. Some of these RST packets are sent from "regular" TCP processing (e.g. outside the sequence window) and others from non-open ports. Making these distinctable by carrying different markings - depending which code path was sending them - would give clues as to what ports/services may be reachable, and give rise to more targetted attacks.
As part if this change, when introducing this new capability (modified TCP header markings for sessions), the irregular codepath is addressed too, such that a 3rd party can not deduct which of the two different paths was sending out the RST, reducing an information side-channel.


Gotcha, thanks. Would "This value also uses ECN for RST replies to probes of non-open ports." mean the same? It looks clearer to me.

  • rebase
  • make tcp_ecn_get_ace static inline
  • include accecn for ecn++
  • fix ecn bits in timewait
  • fix syn-sent ip_ecn marking

Mnor wording nit, can be fixed on commit.

rscheff marked an inline comment as done.
  • update man page wording
  • include ect0 for tcp_respond()

Manual page unchanged besides the bumped Dd.

  • rebase
  • bump man date
  • remove duplicate code

Was removing this intended as part of this review?


I'm confused, there is no change here on; it may be that this diff was uploaded just prior to a flurry of tcp related (including man page) changes, which I have not yet rebased the patch to...

  • rebase
  • use ect1 if session is using that
  • use switch statement instead of if else