Hello,
I was debugging what I thought was a zfs performance issue in KVM I had with FreeBSD 13.0-RC[0-4] and it turned out it wasn't a zfs / openzfs problem but a timecounter thing.
I initially looked at all tweakable settings of zfs, tried to enable scan_legacy because it was slightly better, used dtrace to create FlameGraphs and I saw a lot of calls to time counter functions.
In KVM, I ended up enabling hpet but it made it worse: with HPET on and scan_legacy=1, scrubbing my zroot took 2min53sec and with HPET off (FreeBSD would use ACPI-fast instead), it took only 2min19sec.
I got FlameGraphs of that and saw lots of calls to hpet_get_timecount. I then started to read about clock support in KVM and found kvmclock is supposed to be fastest.
That led me to these old patches by [[ https://lists.freebsd.org/pipermail/freebsd-arch/2015-January/016587.html | Bryan Venteicher ]] and the same thing got submitted again by [[ https://reviews.freebsd.org/D16306 | Stephen J. Kiernan ]] but never progressed.
Adding kvmclock support makes a huge difference (everything I figured out is in this long [[ https://lists.freebsd.org/pipermail/freebsd-fs/2021-March/028894.html | thread ]].
The zroot of my KVM machine can be scrubbed in around 10sec instead of 1min35sec (so that's 9.5 faster).
As for my raidz2 pool (it's made of 8 x 4 TB plus cache and log ssds):
- scrubbing used to take around 23 hours best case (last try took 23:04:17) with kvmclock it's around 12 hours (12:33:11): that's almost twice as fast
- resilvering one drive took 1 days 10:31:23 and with kvmclock it was around 21 hours (21:21:25): that's 1.7 time faster
This is a simple modification which yields a huge performance boost.
Regards,
Mathieu