I was debugging what I thought was a zfs performance issue in KVM I had with FreeBSD 13.0-RC[0-4] and it turned out it wasn't a zfs / openzfs problem but a timecounter thing.
I initially looked at all tweakable settings of zfs, tried to enable scan_legacy because it was slightly better, used dtrace to create FlameGraphs and I saw a lot of calls to time counter functions.
In KVM, I ended up enabling hpet but it made it worse: with HPET on and scan_legacy=1, scrubbing my zroot took 2min53sec and with HPET off (FreeBSD would use ACPI-fast instead), it took only 2min19sec.
I got FlameGraphs of that and saw lots of calls to hpet_get_timecount. I then started to read about clock support in KVM and found kvmclock is supposed to be fastest.
That led me to these old patches by Bryan Venteicher and the same thing got submitted again by Stephen J. Kiernan but never progressed.
Adding kvmclock support makes a huge difference (everything I figured out is in this long thread.
The zroot of my KVM machine can be scrubbed in around 10sec instead of 1min35sec (so that's 9.5 faster).
As for my raidz2 pool (it's made of 8 x 4 TB plus cache and log ssds):
- scrubbing used to take around 23 hours best case (last try took 23:04:17) with kvmclock it's around 12 hours (12:33:11): that's almost twice as fast
- resilvering one drive took 1 days 10:31:23 and with kvmclock it was around 21 hours (21:21:25): that's 1.7 time faster
This is a simple modification which yields a huge performance boost. The patch applies cleanly on origin/releng/13.0