Differential D20839

Use unmapped (M_NOMAP) mbufs for zero-copy AIO writes via TOE.
ClosedPublic
Actions

Authored by jhb on Jul 2 2019, 11:16 PM.

Details

Reviewers

np
gallatin

Commits

rS349649: Use unmapped (M_NOMAP) mbufs for zero-copy AIO writes via TOE.

Summary

Previously the TOE code used its own custom unmapped mbufs via
EXT_FLAG_VENDOR1. The old version always wired the entire AIO request
buffer first for the duration of the AIO operation and constructed
multiple mbufs which used the wired buffer as an external buffer.

The new version determines how much room is available in the socket
buffer and only wires the pages needed for the available room building
chains of M_NOMAP mbufs. This means that a large AIO write will now
limit the amount of wired memory it uses to the size of the socket
buffer.

Diff Detail

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 25190
Build 23872: arc lint + arc unit

Event Timeline

jhb created this revision.Jul 2 2019, 11:16 PM

Herald added a subscriber: ae. · View Herald TranscriptJul 2 2019, 11:16 PM

Harbormaster completed remote builds in B25184: Diff 59332.Jul 2 2019, 11:16 PM

I added Mark to be aware of another place using held pages that the pending wire count patch would touch. Drew in case you are curious as this is another use for M_NOMAP.

sys/dev/cxgbe/tom/t4_cpl_io.c
1944	This is effectively vm_unhold_pages(), but it has to use PHYS_TO_VM_PAGE to get back to the page pointer.

In D20839#451257, @jhb wrote:

I added Mark to be aware of another place using held pages that the pending wire count patch would touch.

Thanks, this will be trivial to convert.

gallatin accepted this revision.Jul 3 2019, 1:26 PM

gallatin added inline comments.

sys/dev/cxgbe/tom/t4_cpl_io.c
2020	Why are you using arg2 rather than arg1? Is there a use of arg1 that I'm missing? In general, you want to prefer arg1, as it falls inside the 2nd cacheline in the mbuf, but arg2 will cause you to touch the 3rd cache line and incur a miss.