netmap: don't schedule kqueue notify task when kqueue is not used

[vmaffione]

Summary

This change adds a counter (kqueue_users) to keep track of how many kqueue users are referencing a given struct nm_selinfo.
In this way, nm_os_selwakeup() can schedule the kevent notification task only when kqueue is actually being used.
This is important to avoid wasting CPU in the common case where kqueue is not used.

Test Plan

Two bhyve VMs over VALE.
Local testing kqueue usage.

[afedorov]

sys/dev/netmap/netmap_freebsd.c

1355 ↗

(On Diff #53868)

Maybe we can use the knlist_empty (9) function to check if kqueue is used?

Something like:

mtx_lock(&si->m);
if (!knlist_empty(&si->si.si_note)) {
       mtx_unlock(&si->m);
       taskqueue_enqueue(si->ntfytq, &si->ntfytask);
} else {
       mtx_unlock(&si->m);
}

[vmaffione]

We can't. We use a taskqueue there precisely because we cannot take the si->m lock (or we get LOR).

[afedorov]

Oops, I forgot this. I will test the current patch.

[vmaffione]

Thanks.
In my tests I can see that without the patch, two pkt-gen communicating over a VALE port will consume 9% CPU on kqueue processing (although kqueue is not used), on a separate CPU.
With the patch, the 9% goes away.

[afedorov]

Sorry for the long delay. My tests didn't find any regression.

[vmaffione]

No worries, and thanks for testing!