increase libmd buffer size to 1m

[lifanov]

Summary

Current buffer size leads to unusable performance with ZFS record size of 1m.

Before:

$ /usr/bin/time /sbin/md5 /usr/ports/distfiles/t-engine4-src-1.5.5.tar.bz2
MD5 (/usr/ports/distfiles/t-engine4-src-1.5.5.tar.bz2) = f40f9493fb327bade5f182c958df2aba
      334.85 real         0.94 user         2.21 sys

After:

$ /usr/bin/time /sbin/md5 /usr/ports/distfiles/t-engine4-src-1.5.5.tar.bz2
MD5 (/usr/ports/distfiles/t-engine4-src-1.5.5.tar.bz2) = f40f9493fb327bade5f182c958df2aba
       10.13 real         0.78 user         0.08 sys

Test Plan

run md5, sha256, skein512, etc.

[allanjude]

I am confused by your results:

SHA512 (16k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = 7ebfc0b7e503b088bda5c351abe0643a8c11d7c5a7da2b3225135f6fb0afae148fd0a3b5eb8c8c21bed41b1991b2656cb68ecd60309220239c81ecd3cf337fd4

3.93 real         3.30 user         0.63 sys

SHA512 (64k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = 7ebfc0b7e503b088bda5c351abe0643a8c11d7c5a7da2b3225135f6fb0afae148fd0a3b5eb8c8c21bed41b1991b2656cb68ecd60309220239c81ecd3cf337fd4

3.86 real         3.46 user         0.40 sys

SHA512 (128k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = 7ebfc0b7e503b088bda5c351abe0643a8c11d7c5a7da2b3225135f6fb0afae148fd0a3b5eb8c8c21bed41b1991b2656cb68ecd60309220239c81ecd3cf337fd4

3.85 real         3.18 user         0.66 sys

SHA512 (1024k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = 7ebfc0b7e503b088bda5c351abe0643a8c11d7c5a7da2b3225135f6fb0afae148fd0a3b5eb8c8c21bed41b1991b2656cb68ecd60309220239c81ecd3cf337fd4

3.89 real         3.19 user         0.69 sys

MD5 (16k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = bec28c9efd8392b1418b5655d6c15bb9

2.90 real         2.28 user         0.62 sys

MD5 (64k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = bec28c9efd8392b1418b5655d6c15bb9

2.84 real         2.36 user         0.47 sys

MD5 (128k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = bec28c9efd8392b1418b5655d6c15bb9

2.84 real         2.35 user         0.49 sys

MD5 (1024k/FreeBSD-12.0-CURRENT-amd64-20170912-r323499-memstick.img) = bec28c9efd8392b1418b5655d6c15bb9

2.89 real         2.38 user         0.50 sys

[allanjude]

I think something else is going on. Can you try the with and without with 'truss -c' instead of 'time'

[lifanov]

Sure.

Before:

syscall                     seconds   calls  errors
cap_enter               0.000007683       1       0
cap_ioctls_limit        0.000013340       2       0
__sysctl                0.000033176       4       0
issetugid               0.000012642       2       0
write                   0.000014249       1       0
sigprocmask             0.000104417      16       0
readlink                0.000010616       1       1
read                  328.301444459   25719       0
openat                  0.000086678       6       0
open                    0.000028566       2       0
munmap                  0.000052800       5       0
mprotect                0.000022979       3       0
mmap                    0.000296068      31       0
lseek                   0.000007055       1       0
getdirentries           0.000055177       2       0
fstatfs                 0.000009918       1       0
fstatat                 0.000123692      11       0
fstat                   0.000049381       6       0
close                   0.000054059       7       0
cap_rights_limit        0.000020324       3       0
cap_getmode             0.000006426       1       0
cap_fcntls_limit        0.000012362       2       0
access                  0.000021023       2       0
                      ------------- ------- -------
                      328.302487090   25829       1

After:

syscall                     seconds   calls  errors
cap_enter               0.000007823       1       0
cap_ioctls_limit        0.000014528       2       0
__sysctl                0.000036179       4       0
issetugid               0.000012431       2       0
write                   0.000010966       1       0
sigprocmask             0.000105815      16       0
readlink                0.000011105       1       1
read                    9.412589033     405       0
openat                  0.000092964       6       0
open                    0.000033246       2       0
munmap                  0.000064467       5       0
mprotect                0.000024516       3       0
mmap                    0.000327432      31       0
lseek                   0.000007403       1       0
getdirentries           0.000056853       2       0
fstatfs                 0.000011595       1       0
fstatat                 0.000139130      11       0
fstat                   0.000052245       6       0
close                   0.000056924       7       0
cap_rights_limit        0.000021582       3       0
cap_getmode             0.000006426       1       0
cap_fcntls_limit        0.000013550       2       0
access                  0.000023608       2       0
                      ------------- ------- -------
                        9.413719821     515       1

[lifanov]

Allan helped me find the mistake (I had primarycache=metadata set on this dataset).
With this addressed, the performance gain from switching to 1m buffer is minimal.