Change Details

This patch adds a sysctl to control whether rx refills are limited by a budget parameter or not. The default preserves the existing behavior. It is not clear what the perceived benefit of limiting receive descriptor refills is. Except for a startup transient due to the limited initial filling of receive rings during init, under load and without any other type of limit imposed, the refill work would be naturally limited according to the receive packet processing budget because the only descriptors available to be refilled would be the ones just released by the retrieving the current batch of packets. It is further not clear, even if an argument can be successfully made as to what the purpose of the limit is, why the refill limit is the packet-reception limit (plus an arbitrary constant). Packets are made of fragments, fragments are 1:1 with driver ring descriptors, and driver ring descriptors are what need to be refilled. Limiting refills based on the packet limit misses an entire dimension of the actual resource consumption (number of fragments per packet) and is guaranteed to fall behind (resulting in starvation and dropped packets) whenever there is more than a little traffic with multiple fragments per packet, except where coincidences favorably align. Concretely, simple iperf testing with the vmx driver configured with device LRO enabled (multiple fragments per packet) demonstrates that the existing refill-limiting behavior results in dropped packets and higher CPU load (because additional receive interrupts must be processed that are refill-only work) for the same TCP throughput. Turning off the refill limit via the new sysctl eliminates dropped packets and reduces CPU consumption during the test.

This patch removes the limit on rx refills. It is not clear what the perceived benefit of limiting receive descriptor refills is. Except for a startup transient due to the limited initial filling of receive rings during init, under load and without any other type of limit imposed, the refill work would be naturally limited according to the receive packet processing budget because the only descriptors available to be refilled would be the ones just released by the retrieving the current batch of packets. It is further not clear, even if an argument can be successfully made as to what the purpose of the limit is, why the refill limit is the packet-reception limit (plus an arbitrary constant). Packets are made of fragments, fragments are 1:1 with driver ring descriptors, and driver ring descriptors are what need to be refilled. Limiting refills based on the packet limit misses an entire dimension of the actual resource consumption (number of fragments per packet) and is guaranteed to fall behind (resulting in starvation and dropped packets) whenever there is more than a little traffic with multiple fragments per packet, except where coincidences favorably align. Concretely, simple iperf testing with the vmx driver configured with device LRO enabled (multiple fragments per packet) demonstrates that the existing refill-limiting behavior results in dropped packets and higher CPU load (because additional receive interrupts must be processed that are refill-only work) for the same TCP throughput. Removing the refill limit eliminates dropped packets and reduces CPU consumption during the test.

This patch adds a sysctl to control whether rx refills are limited by a budget parameter or not. The default preserves the existing behaviorremoves the limit on rx refills. It is not clear what the perceived benefit of limiting receive descriptor refills is. Except for a startup transient due to the limited initial filling of receive rings during init, under load and without any other type of limit imposed, the refill work would be naturally limited according to the receive packet processing budget because the only descriptors available to be refilled would be the ones just released by the retrieving the current batch of packets. It is further not clear, even if an argument can be successfully made as to what the purpose of the limit is, why the refill limit is the packet-reception limit (plus an arbitrary constant). Packets are made of fragments, fragments are 1:1 with driver ring descriptors, and driver ring descriptors are what need to be refilled. Limiting refills based on the packet limit misses an entire dimension of the actual resource consumption (number of fragments per packet) and is guaranteed to fall behind (resulting in starvation and dropped packets) whenever there is more than a little traffic with multiple fragments per packet, except where coincidences favorably align. Concretely, simple iperf testing with the vmx driver configured with device LRO enabled (multiple fragments per packet) demonstrates that the existing refill-limiting behavior results in dropped packets and higher CPU load (because additional receive interrupts must be processed that are refill-only work) for the same TCP throughput. Turning offRemoving the refill limit via the new sysctlit eliminates dropped packets and reduces CPU consumption during the test.