hyperv/hn: Improve sending performance

[sepherosa_gmail.com]

Summary

• Avoid main lock contention by trylock for if_start, if that fails, schedule TX taskqueue for if_start
• Don't do direct sending if the packet to be sent is large, e.g. TSO packet.

This change gives me stable 9.1Gbps TCP sending performance w/ TSO over a 10Gbe directly connected network (the performance fluctuated between 4Gbps and 9Gbps before this commit). It also improves non-TSO TCP sending performance a lot.

[royger]

Out of curiosity, those 9.1Gbps are with a debug kernel or a non-debug one?

[sepherosa_gmail.com]

In D5074#107911, @royger wrote:

Out of curiosity, those 9.1Gbps are with a debug kernel or a non-debug one?

Kernel w/o WITNESS, but w/ INVARIANTS. Make options has DEBUG=-g.

[royger]

sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c

399 ↗

(On Diff #12702)

Do you really need to use the _fast versions of the taskqueue? This is only needed if you are adding tasks from a fast interrupt filter (ie: not a handler) or already holding a spinlock.

[sepherosa_gmail.com]

sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c

399 ↗

(On Diff #12702)

I believe the taskq contention could be high, since many threads will try enqing start task. And well, I don't think it hurts to use a fast taskqueue here.

[royger]

sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c

399 ↗

(On Diff #12702)

Yes, I think it's fine, just was curios because I wasn't able to spot it using any fast interrupt handlers.

IMHO, it is also worth keeping into account that on virtualized systems there's a great chance that not all vCPUs run concurrently (depending on whether the hypervisor implements gang-scheduling), so spinlocks can introduce quite a lot of contention because the holder of the spinlock might not be scheduled, and another vCPU might be spinning to get a hold of the lock.

[sepherosa_gmail.com]

sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c

399 ↗

(On Diff #12702)

It is an interesting point of view; I will note it :). I am also going to experiment some other ways, which may not cause contentions at all on driver tx lock or taskqueue lock.