I like the idea of increasing the MTU. Is there a reason why you select 0xffff? This is the maximum size of an IPv4 packet, but an IPv6 packet can have a size of 0xffff + 40. Why not use that value? Would it make sense to use a multiple of 4?

In D59450#1363624, @tuexen wrote:

I like the idea of increasing the MTU. Is there a reason why you select 0xffff? This is the maximum size of an IPv4 packet, but an IPv6 packet can have a size of 0xffff + 40. Why not use that value? Would it make sense to use a multiple of 4?

I didn't measure, but speculatively I decided that it would be better to have it a multiple of PAGE_SIZE rather than multiple + tiny. In the second case a bulk transfer would allocate an extra page (or an mbuf, if optimized) that would be underutilized.

In D59450#1363646, @glebius wrote:

In D59450#1363624, @tuexen wrote:

I like the idea of increasing the MTU. Is there a reason why you select 0xffff? This is the maximum size of an IPv4 packet, but an IPv6 packet can have a size of 0xffff + 40. Why not use that value? Would it make sense to use a multiple of 4?

I didn't measure, but speculatively I decided that it would be better to have it a multiple of PAGE_SIZE rather than multiple + tiny. In the second case a bulk transfer would allocate an extra page (or an mbuf, if optimized) that would be underutilized.

That makes sense. But don't you reserve always some additional space at the beginning of the mbuf chain for the link layer header? I also tried to set the MTU to a value larger than 65535 using ifconfig and it failed. This is because in if.c there is a check that the MTU does not exceed IF_MAXMTU.

So I suggest to use IF_MAXMTU as the default MTU instead of IP_MAXPACKET.

@glebius I think this is not a breaking change, but it improves the performance significantly ( now with a larger default MTU ). Does it deserve a change log or release note ?

In D59450#1370334, @zlei wrote:

Some time ago I was wondering why the MTU of loopback interface was limited to 16K, I thought a larger one should have better performance so I did a simple test ( iperf3 on lo0 ). It was weird that I saw performance degrees when increasing the MTU to 64K.

It appears the performance degree no longer happens. Retested with different MTUs.

MTU 16K

# ifconfig lo0 mtu 16384
# iperf3 -c 127.0.0.1
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  63.4 GBytes  54.4 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  63.4 GBytes  54.4 Gbits/sec                  receiver

MTU 32K

# ifconfig lo0 mtu 32768
# iperf3 -c 127.0.0.1
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  85.2 GBytes  73.2 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  85.2 GBytes  73.2 Gbits/sec                  receiver

MTU 64K

# ifconfig lo0 mtu 65535
# iperf3 -c 127.0.0.1
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  85.1 GBytes  72.8 Gbits/sec    0            sender
[  5]   0.00-10.04  sec  85.1 GBytes  72.8 Gbits/sec                  receiver

Tested with 14.5-RELEASE. The hardware is 4 core Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz with 16GiB memory.

For the case of MTU=64KB, in FreeBSD you have to change the net.inet.tcp.sendspace value from default 32KB to 64KB or larger to make sense of using MSS=64KB. If so, you also have to increase the net.inet.tcp.recvspace value from default 64KB to a larger value.

Reference:
https://wiki.freebsd.org/chengcui/tcp_loopback_perf

In D59450#1370998, @cc wrote:

For the case of MTU=64KB, in FreeBSD you have to change the net.inet.tcp.sendspace value from default 32KB to 64KB or larger to make sense of using MSS=64KB. If so, you also have to increase the net.inet.tcp.recvspace value from default 64KB to a larger value.

Reference:
https://wiki.freebsd.org/chengcui/tcp_loopback_perf

Thanks for the hints! That is extremely useful !

So I did another round of test. To minimal the impact of sendspace and recvspace, I bumped both of them to 128KB. For the iperf3 client, I added two options --zerocopy and --affinity to minimize the overhead of syscall and scheduling.

# sysctl net.inet.tcp | grep space
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 131072

The TCP congestion algorithm is cubic .

# sysctl -n net.inet.tcp.cc.algorithm
cubic

MTU 16K

# ifconfig lo0 mtu 16384
# iperf3 -c 127.0.0.1 --verbose --zerocopy --affinity 1,3
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  61.6 GBytes  52.9 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  61.6 GBytes  52.9 Gbits/sec                  receiver
CPU Utilization: local/sender 36.0% (2.8%u/33.3%s), remote/receiver 66.2% (4.3%u/62.0%s)

MTU 32K

# ifconfig lo0 mtu 32768
# iperf3 -c 127.0.0.1 --verbose --zerocopy --affinity 1,3
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   108 GBytes  93.0 Gbits/sec    0            sender
[  5]   0.00-10.00  sec   108 GBytes  93.0 Gbits/sec                  receiver
CPU Utilization: local/sender 70.1% (4.2%u/65.8%s), remote/receiver 91.2% (2.4%u/88.7%s)

MTU 64K

# ifconfig lo0 mtu 65535
# iperf3 -c 127.0.0.1 --verbose --zerocopy --affinity 1,3
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   118 GBytes   101 Gbits/sec    0            sender
[  5]   0.00-10.02  sec   118 GBytes   101 Gbits/sec                  receiver
CPU Utilization: local/sender 76.0% (5.3%u/70.8%s), remote/receiver 100.0% (1.7%u/98.2%s)

Thanks for testing, guys! This inspires a change where we would use larger send/recv space for local connections ignoring the sysctl. I will post that separately.