Page MenuHomeFreeBSD

iflib: Use a bounded buf_ring for simple_tx
Needs ReviewPublic

Authored by gallatin on Mon, Aug 17, 11:02 PM.
Tags
None
Referenced Files
F171791840: D58901.diff
Sun, Sep 13, 1:38 PM
F171729558: D58901.diff
Sun, Sep 13, 2:19 AM
F171729320: D58901.diff
Sun, Sep 13, 2:18 AM
Unknown Object (File)
Thu, Sep 10, 8:53 PM
Unknown Object (File)
Tue, Sep 8, 4:32 AM
Unknown Object (File)
Mon, Sep 7, 7:42 PM
Unknown Object (File)
Mon, Sep 7, 3:16 AM
Unknown Object (File)
Sat, Sep 5, 5:53 PM
Subscribers

Details

Summary

Implement buf_ring/drbr deferred transmit in iflib. This is intended
to allow the new simpler code path to replace mp_ring. This patch
makes the simple_tx outperform mp_ring by a wide margin when CPU is
the bottleneck (eg, cannot fill the NIC). See graphs at:
https://people.freebsd.org/~gallatin/mpring_vs_simple_tx

One thing I noticed while developing this patch is that a simple mutex
with no deferral generally outperformed both mp_ring and drbr at high
levels of contention for the same queue. This is inherent in a
bounded MPSC queue where multiple producers are contending on claiming
ring entries. So I came up with the idea of bounding the number of
producers such that the deferral ring would devolve to a mutex when
contention was high.

Identifying the crossover point in a general way was hard. On
different machines, the point between a mutex and a deferral ring was
very different and also depended on the placement of the producers. I
eventually realized that on the large AMD EPYC servers that I was
testing on, the crossover point generally coincided with the work
spilling into another CCX. So I developed an approach where we limit
the number of producers by limiting simultanious producers to the same
L3 AND by bounding the number of simultanious producers.

This limit can be adjusted via the sysctl net.iflib.max_producers.

The number of packets drained from the deferral ring is limited
by net.iflib.simple_drain_quota. The intent is to process just
enough packets in the gtaskq context so as to make space to
allow threads to make progress. The task also uses a trylock
so as to avoid blocking, waiting for a thread to drain.

This change also enables ALTQ support for simple tx.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

This comment was removed by gallatin.
sys/net/iflib.c
163–164

I think this may need #ifdef ALTQ or some other handling to avoid build breakage where that is undefined

4304

This wont be reachable with the same check right above taking the goto, rebase issue?

8182

for ALTQ this may need (void)iflib_completed_tx_reclaim(txq, NULL); before iflib_simple_drbr_drain because it gives up silently under lock contention

Some preliminary testing on a pedestrian setup: DUT is network limited, 1Gbps and 10Gbps interfaces both tested with 1 MSI-X q. iperf3 off q0 cpu on both sides. Each phase used three 10second repetitions at 1, 2, 4, and 8 parallel senders. Traffic was 1342-byte UDP with an unlimited offered rate, a 2 MiB socket window, fixed source ports, and fixed client affinity on CPUs 1,3,5,7. The tables contain medians. "Goodput" is receiver bytes normalized to the sender interval. This avoids counting iperf's post send socket drain interval. "Drops" means full software TX-ring enqueue failures: r_drops for A1/A2 and drbr_drops for B.

The one stream ice result seems like the best thing to chase first, any thoughts on that? The context switches to on ice 1 stream and I need to figure out what the 82754 awkwardness is for parallel streams.

ice(4) / E810

Reproducibility disclosure, D58910 is also applied

StreamsGoodput A1 / B / A2 (Gb/s)System CPU A1 / B / A2Context switches/s A1 / B / A2TX enqueue drops/run A1 / B / A2
19.526 / 9.531 / 9.51710.4 / 20.5 / 10.2%0.3k / 217k / 0.3k0 / 0.144m / 0
29.531 / 9.533 / 9.53425.0 / 30.7 / 24.7%843k / 506k / 923k6.36m / 1.79m / 6.52m
49.512 / 9.527 / 9.51347.5 / 54.8 / 47.1%1.11m / 661k / 1.13m28.55m / 2.99m / 28.49m
89.524 / 9.533 / 9.50747.2 / 54.9 / 47.1%1.04m / 619k / 1.08m27.92m / 2.27m / 27.83m

At P2-P8, simple_tx:

  • preserves line-rate throughput
  • reduces context switches by approximately 40-43%
  • reduces full-ring enqueue failures by approximately 72-92%
  • lowers peer CPU consumption and peer socket loss
  • costs approximately 6-8 additional sender system CPU%

em(4) / 82574L

StreamsGoodput A1 / B / A2 (Gb/s)System CPU A1 / B / A2Context switches/s A1 / B / A2TX enqueue drops/run A1 / B / A2
10.956 / 0.955 / 0.95613.2 / 21.6 / 13.7%225k / 38.9k / 231k2.40m / 12.88m / 2.16m
20.956 / 0.955 / 0.95632.8 / 31.7 / 32.7%54.9k / 81.5k / 49.7k17.02m / 22.68m / 17.34m
40.956 / 0.955 / 0.95653.3 / 54.4 / 53.7%11.6k / 104k / 11.1k36.97m / 26.61m / 36.85m
80.956 / 0.955 / 0.95653.4 / 55.2 / 53.6%13.0k / 104k / 10.4k37.32m / 26.23m / 36.59m

At P4/P8 it reduces full-ring enqueue failures by approximately 28%, but increases context switches by roughly 8-10x. At P1 it adds about 8 system CPU% and increases enqueue failures by more than 5x.

gallatin marked 2 inline comments as done.

Updated to 2nd version of the patch. Will update description

gallatin retitled this revision from iflib: implement drbr for simple_tx. to iflib: Use a bounded buf_ring for simple_tx.Thu, Sep 10, 4:27 PM
gallatin edited the summary of this revision. (Show Details)

Retested on the same setup: i7-6700, four cores/eight threads, 1Gbps and 10Gbps interfaces, each with one hardware RX/TX queue. Both endpoints are running without WITNESS. This remains a short, network-limited test on a single LLC machine; it does not exercise cross-LLC admission or the high-bandwidth, many-core case. The changed kernel base and peer topology also mean this is not a bisect of differences from the August results.

A1/A2 are mp_ring; B is simple_tx with the new default settings. Each phase has three 10-second runs at 1, 2, 4, and 8 senders. Traffic is 1342-byte UDP, unlimited offered rate, 2 MiB socket buffers, fixed source ports, and affinity to CPUs 1,3,5,7. Queue/IRQ placement and flow identities were checked across phases.

The tables show medians in A1 / B / A2 order. Goodput is received bytes divided by the sender interval, excluding the receiver's final socket-drain time. Drops are software-ring enqueue failures: r_drops for A and drbr_drops for B. System CPU is the whole DUT's percentage.

ice(4) / E810

D58910 remains applied in all phases. The test kernel is based on main cb2964f840e8.

StreamsGoodput A1 / B / A2 (Gb/s)System CPU A1 / B / A2 (%)Context switches/s A1 / B / A2TX enqueue drops/run A1 / B / A2
19.000 / 9.083 / 9.09110.6 / 12.8 / 10.8168k / 844k / 172k1.50m / 0 / 1.41m
29.521 / 9.520 / 9.52222.8 / 26.9 / 23.1798k / 1.18m / 798k11.56m / 2.85m / 11.37m
47.085 / 9.532 / 7.26946.0 / 51.7 / 45.9877k / 1.19m / 884k33.57m / 3.71m / 33.56m
87.290 / 9.530 / 7.28945.3 / 52.1 / 45.4836k / 1.13m / 825k32.39m / 2.07m / 32.63m

At P4/P8, simple_tx:

  • Improves received goodput by approximately 31% versus A2.
  • Reduces receiver loss from approximately 24–26% to 0% / 0.04%.
  • Substantially reduces full-ring enqueue failures.
  • Costs approximately 6–7 additional sender system CPU percentage points.
  • Increases context switches by approximately 34–37%, unlike the reduction in my earlier results.

Both modes already transmit about 9.53 Gb/s at P4/P8. The goodput difference corresponds to input drops on the peer, which return in A2. That is reproducible, but I have not isolated why the receiver handles the two transmit patterns differently.

P1 reception varies with overlapping ranges in all three phases; there is no clear throughput improvement there. P2 remains about 9.52 Gb/s.

em(4) / 82574L

StreamsGoodput A1 / B / A2 (Gb/s)System CPU A1 / B / A2 (%)Context switches/s A1 / B / A2TX enqueue drops/run A1 / B / A2
10.956 / 0.955 / 0.95614.1 / 13.3 / 13.3233k / 1.19m / 224k2.08m / 10.22m / 2.22m
20.956 / 0.955 / 0.95632.7 / 29.3 / 33.169.8k / 561k / 46.7k13.34m / 20.45m / 15.85m
40.956 / 0.955 / 0.95654.2 / 52.9 / 54.29.0k / 535k / 9.5k36.21m / 22.53m / 36.18m
80.956 / 0.955 / 0.95754.8 / 52.2 / 54.49.9k / 675k / 14.0k36.16m / 18.19m / 36.01m

Line-rate delivery is preserved without receiver loss. Sender system CPU is similar or slightly lower with simple_tx, but context switches increase considerably. Enqueue failures increase at P1/P2 and decrease at P4/P8.

Paced follow-up

The unlimited tests also measure how quickly each mode rejects excess work, so I repeated single-stream tests below saturation. Three runs each, 100us pacing, B followed by A2. These cells are A2 / B, not A1 / B / A2.

Interface / offered rateGoodput A2 / B (Gb/s)System CPU A2 / B (%)Context switches/s A2 / B
em / 850 Mb/s0.850 / 0.8502.7 / 2.3163k / 164k
ice / 5 Gb/s5.000 / 5.0007.3 / 6.7315k / 379k

Neither mode has enqueue drops or receiver loss in these paced runs. The large em context-switch difference disappears below saturation. Ice still shows about 20% more context switches, with slightly lower system CPU.