Use unmapped (M_NOMAP) mbufs for zero-copy AIO writes via TOE.

[jhb]

Summary

Previously the TOE code used its own custom unmapped mbufs via
EXT_FLAG_VENDOR1. The old version always wired the entire AIO request
buffer first for the duration of the AIO operation and constructed
multiple mbufs which used the wired buffer as an external buffer.

The new version determines how much room is available in the socket
buffer and only wires the pages needed for the available room building
chains of M_NOMAP mbufs. This means that a large AIO write will now
limit the amount of wired memory it uses to the size of the socket
buffer.

Sponsored by: Chelsio

Test Plan

• tested with netperf -a

[jhb]

I added Mark to be aware of another place using held pages that the pending wire count patch would touch. Drew in case you are curious as this is another use for M_NOMAP.

sys/dev/cxgbe/tom/t4_cpl_io.c

1944 ↗

(On Diff #59332)

This is effectively vm_unhold_pages(), but it has to use PHYS_TO_VM_PAGE to get back to the page pointer.

[markj]

In D20839#451257, @jhb wrote:

I added Mark to be aware of another place using held pages that the pending wire count patch would touch.

Thanks, this will be trivial to convert.

[gallatin]

sys/dev/cxgbe/tom/t4_cpl_io.c

2020 ↗

(On Diff #59332)

Why are you using arg2 rather than arg1? Is there a use of arg1 that I'm missing?

In general, you want to prefer arg1, as it falls inside the 2nd cacheline in the mbuf, but arg2 will cause you to touch the 3rd cache line and incur a miss.

[jhb]

sys/dev/cxgbe/tom/t4_cpl_io.c

2020 ↗

(On Diff #59332)

No good reason, too focused on sendfile use ext_arg2 and missed it also used arg1 (and that arg1 is the "primary" arg for sendfile). Will fix.

[jhb]

• Use ext_arg1 instead of ext_arg2.