bhyve: add checksum offloading for tap

[timo.voelker_fh-muenster.de]

Summary

To allow the use of checksum offloading for packets to and from a VM, this patch changes the following.

• Set the vnet_hdr_len > 0 for tap.
• Report that the tap backend supports receiving packets with partial checksum (VIRTIO_NET_F_CSUM) and that the guest can get packets with partial checksum (VIRTIO_NET_F_GUEST_CSUM).
• Enable transmission checksum offloading (IFCAP_TXCSUM and IFCAP_TXCSUM) on the tap interface if the feature VIRTIO_NET_F_GUEST_CSUM has been negotiated.

Note that if the VM sends a packet with partial checksum and the host sends it out over an interface that does not support checksum offloading, the packet go out with an incorrect checksum. D56564 moves the check of whether an interface supports checksum offloading close to the driver and prevents this issue.

Test Plan

Create a tap interface.

ifconfig tap0 create

Start a FreeBSD VM with bhyve using this tap interface. The tap0 interface on the host and the vtnet0 interface in the VM should have txcsum and txcsum6 enabled. Configure an IP address on both interfaces.

Host

ifconfig tap0 inet 192.168.0.1/24

VM

ifconfig vtnet0 inet 192.168.0.2/24

Use nc for a test with TCP packets. In the VM run

nc -l 1234

On the host, run

nc 192.168.0.2 1234

Result: Both the host and the VM offload the computation of the TCP checksum. Host and VM can communicate without computing the TCP checksum.

[markj]

usr.sbin/bhyve/net_backends.c

93

This value causes the assertion on lines 429-430 to fail if the frontend negotiates mergeable rx buffers with the guest virtio driver.

[timo.voelker_fh-muenster.de]

Update the vnet_hdr_len in tap_set_cap().

Change the way of enabling transmission checksum offloading (see new summary).

[timo.voelker_fh-muenster.de]

usr.sbin/bhyve/net_backends.c

93

Thanks, that helped. Now, I update the vnet_hdr_len in tap_set_cap().

[cc]

In your test, will this checksum offloading save some CPU cycles. If so, is there any benchmark showing that stats?

[timo.voelker_fh-muenster.de]

In D51688#1378630, @cc wrote:

In your test, will this checksum offloading save some CPU cycles. If so, is there any benchmark showing that stats?

That's a good question. I ran tests some time ago but couldn't find any measurements in my notes. I just ran another test.

I created a tap interface, started a FreeBSD VM with bhyve, and configured the tap interface on the host and the vtnet0 interface in the VM with IP addresses in the same subnet. Then I started iperf3 -s on the host and 20 times iperf3 -c <hostIP> in the VM, once with txcsum and once with txcsum disabled, i.e., after ifconifg vtnet0 -txcsum, and saved the throughput value in Mbit/s in tp2_txcsum.txt and tp2_-txcsum.txt, respectively.

% ministat tp2_-txcsum.txt tp2_txcsum.txt
x tp2_-txcsum.txt
+ tp2_txcsum.txt
+---------------------------------------------------------------------------+
|              x xxx                  + + + +                               |
|x     xxx  xxxxxxxx x*  x+ x++ + + +++ + +++    +         +               +|
|        |______AM____|     |__________MA___________|                       |
+---------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20          1735          1803          1774        1771.8     15.729958
+  20          1786          1919          1830       1831.95     29.357103
Difference at 95.0% confidence
	60.15 +/- 15.0735
	3.39485% +/- 0.857272%
	(Student's t, pooled s = 23.5507)

The throughput is about 60 Mbit/s higher with transmit checksum offloading (txcsum). That's something. I expect a bigger win with TSO and checksum offloading is a requirement for TSO.

[cc]

In D51688#1381014, @timo.voelker_fh-muenster.de wrote:

The throughput is about 60 Mbit/s higher with transmit checksum offloading (txcsum). That's something. I expect a bigger win with TSO and checksum offloading is a requirement for TSO.

Thanks for sharing! That more than 3% improvement looks acceptable. I am also interested to test myself in my own testbed, as I am using vm-bhyve + tap to manage my VMs. Any performance improvement is appreciated.

[cc]

In D51688#1381454, @cc wrote:

In D51688#1381014, @timo.voelker_fh-muenster.de wrote:

The throughput is about 60 Mbit/s higher with transmit checksum offloading (txcsum). That's something. I expect a bigger win with TSO and checksum offloading is a requirement for TSO.

How to enable TSO in the vtnet0 of VM? I can enable TSO in the tap0 of the host. But didn't find out TSO enabled in the VM. And the VM sysctl dev.vtnet.0.txq0 below shows TSO not used in the test traffic.

cc@n2:~ % sudo ifconfig tap0 tso
cc@n2:~ % ifconfig tap0 | egrep TSO
	options=480302<TXCSUM,TSO4,TSO6,LINKSTATE,TXCSUM_IPV6>
cc@n2:~ % 

root@n2fbsd16:~ # ifconfig vtnet0 tso4 tso6
root@n2fbsd16:~ # ifconfig vtnet0 | egrep TSO
root@n2fbsd16:~ #

root@n2fbsd16:~ # sysctl dev.vtnet.0.txq0
dev.vtnet.0.txq0.rescheduled: 0
dev.vtnet.0.txq0.tso: 0
dev.vtnet.0.txq0.csum: 3161387
dev.vtnet.0.txq0.omcasts: 8
dev.vtnet.0.txq0.obytes: 4782003413
dev.vtnet.0.txq0.opackets: 3161404
root@n2fbsd16:~ #

[timo.voelker_fh-muenster.de]

In D51688#1381752, @cc wrote:

In D51688#1381454, @cc wrote:

In D51688#1381014, @timo.voelker_fh-muenster.de wrote:

The throughput is about 60 Mbit/s higher with transmit checksum offloading (txcsum). That's something. I expect a bigger win with TSO and checksum offloading is a requirement for TSO.

How to enable TSO in the vtnet0 of VM? I can enable TSO in the tap0 of the host. But didn't find out TSO enabled in the VM. And the VM sysctl dev.vtnet.0.txq0 below shows TSO not used in the test traffic.

Sorry for not making that clear. Currently, bhyve does not negotiate TSO support. That's why you cannot enable it in the FreeBSD VM. My plan is to add TSO support in the future, and this is a step towards that goal.

By the way, QEMU on Linux negotiates TSO support, and FreeBSD in the VM enables TSO on vtnet by default.

[cc]

In D51688#1382029, @timo.voelker_fh-muenster.de wrote:

In D51688#1381752, @cc wrote:

In D51688#1381454, @cc wrote:

In D51688#1381014, @timo.voelker_fh-muenster.de wrote:

The throughput is about 60 Mbit/s higher with transmit checksum offloading (txcsum). That's something. I expect a bigger win with TSO and checksum offloading is a requirement for TSO.

How to enable TSO in the vtnet0 of VM? I can enable TSO in the tap0 of the host. But didn't find out TSO enabled in the VM. And the VM sysctl dev.vtnet.0.txq0 below shows TSO not used in the test traffic.

Sorry for not making that clear. Currently, bhyve does not negotiate TSO support. That's why you cannot enable it in the FreeBSD VM. My plan is to add TSO support in the future, and this is a step towards that goal.

By the way, QEMU on Linux negotiates TSO support, and FreeBSD in the VM enables TSO on vtnet by default.

Thanks for the clarification. I got similar result with the +3.x% iperf3 throughput gain in GENERIC build testD51688. And I didn't see any problem during the test.