teach ena driver about RSS kernel option
ClosedPublic
Actions

Authored by avg on May 6 2020, 8:45 AM.

Details

Reviewers

mw
adrian

Commits

rG5b8f3948f166: Upgrade ENA to v2.4.0
rS362530: teach ena driver about RSS kernel option

Summary

Networking is broken if the driver configures its (virtual) hardware to use
a hash algorithm (or a key) different from the one that the network stack
(software RSS) uses. This can be seen with connections initiated from the
host. The PCB will be placed into the hash table based on the hash value
calculated by the software. The hardware-calculated hash value in reponse
packets will be different, so the PCB won't be found.

The work is sponsored by Panzura.

Test Plan

Tested with a kernel compiled with 'options RSS' on an instance with ena
driver.

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

avg created this revision.May 6 2020, 8:45 AM

Herald added a subscriber: imp. · View Herald TranscriptMay 6 2020, 8:45 AM

avg requested review of this revision.May 6 2020, 8:45 AM

Harbormaster completed remote builds in B30946: Diff 71455.May 6 2020, 8:45 AM

update the module makefile

Harbormaster completed remote builds in B30961: Diff 71500.May 7 2020, 1:24 PM

idose_amazon.com added a subscriber: idose_amazon.com.May 26 2020, 3:08 PM

pi added a subscriber: pi.May 26 2020, 7:29 PM

This actually looks good to me!

Sigh - this is why when I added the flowid we didn't set the hash for TCP - we relied on the result from the hardware ...

In D24733#550875, @mmacy wrote:

Sigh - this is why when I added the flowid we didn't set the hash for TCP - we relied on the result from the hardware ...

That's like 80% of the problem, the other 20% was getting the pcb into the right group early on and not have to go move it between the RSS buckets.

The flowid stuff is separate from RSS per se. If you use flowid without the microsoft style RSS with the per-CPU work distribution then yeah, you don't need the software hashing. But if you want to have it partition up the work early that also includes putting the pcb into the right RSS hashed pcbgroup, and that requires a software hash.

It's only done, like, once..

In D24733#550888, @adrian wrote:

In D24733#550875, @mmacy wrote:

Sigh - this is why when I added the flowid we didn't set the hash for TCP - we relied on the result from the hardware ...

That's like 80% of the problem, the other 20% was getting the pcb into the right group early on and not have to go move it between the RSS buckets.

The flowid stuff is separate from RSS per se. If you use flowid without the microsoft style RSS with the per-CPU work distribution then yeah, you don't need the software hashing. But if you want to have it partition up the work early that also includes putting the pcb into the right RSS hashed pcbgroup, and that requires a software hash.

It's only done, like, once..

Was the PCBGROUPS stuff actually done? IIRC it relied on netisr changes which were incomplete. I got substantially worse perf turning it on for a number of reasons.

Hi, sorry for late response, missed that previously. I let know about the change to Amazon folks so that they can review it internally as well. I'll update.

dch added a subscriber: dch.Jun 3 2020, 4:06 PM

We managed to get an ack from Amazon, as well.

This revision is now accepted and ready to land.Jun 23 2020, 12:33 AM

adrian accepted this revision.Jun 23 2020, 1:03 AM

Closed by commit rS362530: teach ena driver about RSS kernel option (authored by avg). · Explain WhyJun 23 2020, 4:58 AM

This revision was automatically updated to reflect the committed changes.

avg added a commit: rS362530: teach ena driver about RSS kernel option.

mw added a commit: rG5b8f3948f166: Upgrade ENA to v2.4.0.Oct 12 2021, 2:49 PM