Allow iflib drivers to specify the buffer size used for each receive queue

[pkelsey]

Summary

This patch leaves the existing iflib mechanism for choosing the receive queue buffer size in place, and extends it to allow drivers to specify on a per-queue basis whether iflib should determine the size or a driver-supplied size should be used. 'Per-queue' here uses the iflib terminology where the machinery for a given NIC queue is represented by an iflib queue set, which may itself be composed of multiple iflib queues.

A read-only freelist sysctl is also added to report the buffer size in use for each freelist (and thus for the corresponding receive queue).

The motivation for this change is the vmx driver, for which each NIC receive queue is served by two buffer descriptor rings. The first buffer descriptor ring is used for the first segment of a given packet, and the second buffer descriptor ring is used for any additional segments of a given packet (which cases arises under jumbo frame and LRO configurations). With this receive buffer ring structure and use, there is better memory utilization and throughput performance with jumbo frames and/or LRO when the size of the buffers used in the first descriptor ring are chosen based on the mtu (i.e., by iflib), and the size of the buffers used in the second descriptor ring are always page-sized.

Test Plan

This has been tested on a 12.1 system (with r356932 applied) with em and vmx devices.

This builds for 13-current, but testing on 13-current is waiting for the package system to get fixed.

[erj]

I think this looks okay, but I don't think it'll apply to the Intel drivers, right?

[pkelsey]

In D23947#526398, @erj wrote:

I think this looks okay, but I don't think it'll apply to the Intel drivers, right?

I would expect that the Intel drivers would stick with the existing approach (where the driver updates the max frame size and iflib then chooses the buffer size).