if_lagg: Allow lagg interfaces to be used with NetMap

[thj]

Test Plan

Issues with epair and netmap require hardware to perform testing (as far as I
can tell). Instead I tested with two hosts I have an 4 igc interfaces. I tested
with pkt-gen to verify that traffic can be sent (it could already) and received
by a netmap application:

dut # ifconfig lagg create
lagg0
dut # ifconfig igc1 up
dut # ifconfig igc2 up
dut # sudo ifconfig lagg0 inet 10.4.100.2/24 laggport igc1 laggport igc2 laggproto lacp up

pktgen # ifconfig lagg create
lagg0
pktgen # ifconfig igc0 up
pktgen # ifconfig igc1 up
pktgen # sudo ifconfig lagg0 inet 10.4.100.3/24 laggport igc0 laggport igc1 laggproto lacp up

# cd /usr/obj/code/freebsd/worktrees/netmap/amd64.amd64/tools/tools/netmap
dut # sudo ./pkt-gen -f rx -i lagg0
pktgen # sudo ./pkt-gen -f tx -i lagg0 -n 1000000000 -l 60 -d 198.19.0.1:2000-198.19.0.100 -D a0:36:9f:1e:28:14 -s 198.18.0.1:2000-198.18.0.20 -w 4

[zlei]

I'm not familiar with netmap, just some thoughts.

I've not benchmarked but current approach looks a little heavy weight.
Can we hook netmap directly onto LAGG members so that routine ether_nh_input() can be totally bypassed ?

[markj]

In D37436#851573, @zlei wrote:

I'm not familiar with netmap, just some thoughts.

I've not benchmarked but current approach looks a little heavy weight.
Can we hook netmap directly onto LAGG members so that routine ether_nh_input() can be totally bypassed ?

Do you mean, netmap could override the if_input pointer for member interfaces? I'm not sure if this is really sensible for at least LACP mode since we definitely want the kernel driver to see LACP PDUs even if regular ethernet packets are handed off to the netmap application.

We also have a similar patch for if_bridge. I hacked up an implementation of your suggestion there and didn't find any throughput difference with iperf/TCP, though my current testing environment is quite noisy. if_bridge and if_lagg acquire input packets in ether_input_internal(), so we don't end up skipping that much code in the end. And to implement the suggestion, we have to add an additional indirect function call since we have to pass the lagg/bridge ifp to netmap, not the ifp of the member interface, so an extra dummy if_input routine is needed to switch the ifp pointers.

I think you are right that there is probably some marginal performance improvement to be gained here, but this patch is simpler and is not much more heavy weight in my opinion.

[zlei]

In D37436#857079, @markj wrote:

In D37436#851573, @zlei wrote:

I'm not familiar with netmap, just some thoughts.

I've not benchmarked but current approach looks a little heavy weight.
Can we hook netmap directly onto LAGG members so that routine ether_nh_input() can be totally bypassed ?

Do you mean, netmap could override the if_input pointer for member interfaces? I'm not sure if this is really sensible for at least LACP mode since we definitely want the kernel driver to see LACP PDUs even if regular ethernet packets are handed off to the netmap application.

Ooh, I missed the LACP PDUs, they definitely should be processed by kernel.

So let's go on with this approach.

We also have a similar patch for if_bridge. I hacked up an implementation of your suggestion there and didn't find any throughput difference with iperf/TCP, though my current testing environment is quite noisy. if_bridge and if_lagg acquire input packets in ether_input_internal(), so we don't end up skipping that much code in the end. And to implement the suggestion, we have to add an additional indirect function call since we have to pass the lagg/bridge ifp to netmap, not the ifp of the member interface, so an extra dummy if_input routine is needed to switch the ifp pointers.

Correct me if wrong.
I think NETMAP applications are PPS intensive (maybe also throughput intensive), so iperf/TCP might be not sufficient as the smallest possible payload size should be tested.

I think you are right that there is probably some marginal performance improvement to be gained here, but

this patch is simpler and is not much more heavy weight in my opinion.

I agree.

[markj]

In D37436#857130, @zlei wrote:

In D37436#857079, @markj wrote:

We also have a similar patch for if_bridge. I hacked up an implementation of your suggestion there and didn't find any throughput difference with iperf/TCP, though my current testing environment is quite noisy. if_bridge and if_lagg acquire input packets in ether_input_internal(), so we don't end up skipping that much code in the end. And to implement the suggestion, we have to add an additional indirect function call since we have to pass the lagg/bridge ifp to netmap, not the ifp of the member interface, so an extra dummy if_input routine is needed to switch the ifp pointers.

Correct me if wrong.
I think NETMAP applications are PPS intensive (maybe also throughput intensive), so iperf/TCP might be not sufficient as the smallest possible payload size should be tested.

Certainly this is true, my benchmark was just meant to be a quick sanity check. I will keep my hack around and benchmark further once some bugs are fixed.

I would just note that using netmap in emulated mode is never the best choice for high-performance applications. The main utility of having netmap support in software interfaces like if_lagg is to make it easier to deploy netmap applications on systems with varied network configurations.

[zlei]

Looks good to me. ( Not tested )