Page MenuHomeFreeBSD

dpaa2: Allocate DMA tag per Rx buffer
AcceptedPublic

Authored by dsl on Sun, Sep 6, 8:56 AM.
Tags
None
Referenced Files
F173069525: D59463.diff
Wed, Sep 23, 8:38 AM
F173041934: D59463.id.diff
Wed, Sep 23, 2:46 AM
F173030069: D59463.id186006.diff
Wed, Sep 23, 12:14 AM
F173005911: D59463.id185993.diff
Tue, Sep 22, 7:49 PM
Unknown Object (File)
Mon, Sep 21, 8:39 AM
Unknown Object (File)
Mon, Sep 21, 3:28 AM
Unknown Object (File)
Sat, Sep 19, 11:15 PM
Unknown Object (File)
Sat, Sep 19, 6:22 AM

Details

Reviewers
bz
dch
dsl
Summary

Previously each channel allocated its own set of buffers using the
channel's DMA tag which causes a DMA lock contention under load.
Even a single saturated 1 Gbps link caused ~50,000 adaptive
mutex spin events (as per lockstat) per second. With the proposed
approach there's no more contention on the channel's DMA mutex and the
adaptive mutex spin events dropped to ~6,000/s.

Stress test where iperf3 pushed as much traffic as possible to the 4
ports revealed that throughput drops from expected 940 Mbps down to
600-800 on each link with the "bounce pages lock" generating ~110,000
adaptive mutex spin events per second, but this is to be addressed
later on.

Test Plan
  1. kldload dtraceall
  2. Run iperf3 server
  3. Run iperf3 client on a remote computer and push as much traffic as possible for ~30s
  4. lockstat sleep 30 > ./lockstat_recv.txt

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

dsl requested review of this revision.Sun, Sep 6, 8:56 AM
dsl edited the test plan for this revision. (Show Details)

Pushed to main during the devsummit. It works just fine as far as I can tell and according to the stress tests conducted during the hackathon 202609. Please, complain if it breaks something.

This revision is now accepted and ready to land.Thu, Sep 10, 10:37 AM