IMO it is too special-purpose, and potentially affecting other loads.

Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present. I would prefer that we added infrastructure to assign policy to the file' vm_object, e.g. using some new ioctl on file descriptor, and userspace did the job of explicitly stating the desired policy.

In D20062#431501, @kib wrote:

IMO it is too special-purpose, and potentially affecting other loads.

Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present. I would prefer that we added infrastructure to assign policy to the file' vm_object, e.g. using some new ioctl on file descriptor, and userspace did the job of explicitly stating the desired policy.

I would prefer not to issue an ioctl for every file; that would increase the system call overhead substantially for workloads with small files. Would you be opposed to a sysctl to control the behavior? How about a new sendfile flag?

In D20062#431501, @kib wrote:

IMO it is too special-purpose, and potentially affecting other loads.

Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present. I would prefer that we added infrastructure to assign policy to the file' vm_object, e.g. using some new ioctl on file descriptor, and userspace did the job of explicitly stating the desired policy.

Thinking more about this, I'm having a hard time understanding why you think that this is too special purpose. In this case, the application has told the kernel that he wants the backing pages recycled quickly by using the SF_NOCACHE flag. If that is the case, then we don't expect anybody else to be using those pages, so what is the harm of allocating them in a way which is correct for this application's needs?

I'm going to have to re-test with this

I still do not quite understand the benefits of this change.

Don't you (Netflix) have configurations where machine has at least two netwoks cards, each plugged into lane from separate NUMA domain ? Could it be that you have two tcp connections coming each from its own card and streaming the same file ?

In D20062#432560, @kib wrote:

I still do not quite understand the benefits of this change.

There should be some benefit from just this and D20060 (lacp egress port selection for numa) on machines with more 4 (or more) domains. The combination will at least reduce a non-encrypted sendfile() to a 75% chance of a single domain crossing on DMA write by the storage controllers. Without these 2 changes, then the connection could be local to domain0, the page could be on domain1, and the egress NIC could be on domain 2, for a total of 2 domain crossings for the bulk data (75% chance of a crossing on DMA write from storage, 75% chance of a crossing on DMA read to NIC). With this and D20060, there will be a 75% chance of a single domain crossing for the DMA write, but then the page will be local for the egress NIC. Sadly, the only 4-domain machine I have is AMD, and I have no way to measure Infinity Fabric utilization.

For a dual socket system, without quite a few more patches, the benefit are not going to be very big. That's the problem with NUMA, there are not a lot of measurable incremental improvements unless you have all the pieces in place. However, the following future changes from me, Jeff and John Baldwin make this very important. Specifically

1. My patch to alter the behavior of SO_REUSEPORT_LB listen sockets to filter incoming connections by domain makes it possible for sendfile() to always called by a webserver running on the connection's local domain
2. A patch (done horribly be me, refined by Jeff) to allocate the VM page array and vm_domain (and pa_locks) backed by domain-specific memory make actions to manage the page local
3. The major benefit comes when software kernel TLS is in use, as the page allocated by sendfile() is then read by the CPU as the source for crypto by a TLS worker thread running on the connection's local domain. John is working on upstreaming kernel TLS for us.
4. My patch in D20060 to select lacp egress ports by domain.

Don't you (Netflix) have configurations where machine has at least two netwoks cards, each plugged into lane from separate NUMA domain ? Could it be that you have two tcp connections coming each from its own card and streaming the same file ?

Yes, we are testing configurations with one NIC per NUMA domain. Such configurations are the motivation for this patchset.

At least in our workload, the webserver marks sendfile() requests with SF_NOCACHE when it is serving an "unpopular" file. Eg, files that the data science folks do not anticipate will be streamed again in the near future. Hence it is unlikely that the same file will be bouncing between domains. I was slightly worried about this bouncing effect, and that's why I re-measured things.

The results that I have for UPI cross-domain traffic as reported by Intel's PCM.x as "QPI data traffic/Memory controller traffic" are below.. The system where I'm taking these numbers is a dual-socket Xeon Silver 4216, with 4 NVME and 1 100GbE NIC on each domain, serving 100% TLS encrypted traffic at 180-190Gb/s. This machine is running my entire patch stack, as well as some things from Jeff (UMA cross domain free support).

13%: Original patch (hand plumbed patch to vm_page_alloc_domain_after(), not touching object domain preference): 13%
21%+: Patch as of early yesterday (set domain preference on object only when it is not currently set). Note that the cross-domain traffic seemed to be climbing.
13%: This patch

I think that there is a better approach than what you are attempting here. Once you have the patch that you describe as "My patch to alter the behavior of SO_REUSEPORT_LB listen sockets ...", the thread executing sendfile(2) will be running on the same domain as the socket. Recall Kostik's first comment, "Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present." In other words, if you don't define an object policy, the page allocations will be governed by the thread's policy. A thread policy that says allocate from the local domain should achieve your desired outcome.

In D20062#432787, @alc wrote:

I think that there is a better approach than what you are attempting here. Once you have the patch that you describe as "My patch to alter the behavior of SO_REUSEPORT_LB listen sockets ...", the thread executing sendfile(2) will be running on the same domain as the socket. Recall Kostik's first comment, "Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present." In other words, if you don't define an object policy, the page allocations will be governed by the thread's policy. A thread policy that says allocate from the local domain should achieve your desired outcome.

Brilliant. I feel so silly.

I had not put the thread affinity together, as the SO_REUSEPORT_LB change I made is quite recent, and I've had the sendfile() patch for nearly a year. Thank you so much for the insight, and sorry for the waste of time.