Increase bhyve BLOCKIF_IOV_MAX and VTBLK_RINGSZ to 128 to support modern Windows guests

[richard_meric.id.au]

Summary

Windows 10 and Server 2016+ vms both fail if configured with virtio-blk disk devices due to IOV_MAX limitations in bhyve.
This patch increases this limit as seen in https://reviews.freebsd.org/D10581 but also increases the VTBLK_RINGSZ to avoid breaking Linux guests.

It has been discussed here before:
http://freebsd.1045724.x6.nabble.com/pci-virtio-block-c-Assertion-failed-line-216-td6155786.html
https://reviews.freebsd.org/D10581
https://reviews.freebsd.org/D9033

Note: This patch is derived from a similar fix from the smartos.org project (https://smartos.org/bugview/OS-7175)

[richard_meric.id.au]

Fix cut-and-paste error in patch

[araujo]

Thanks for the patch, I will take a look on within the next two weeks.

[pmooney_pfmooney.com]

This is effectively what we did in SmartOS/illumos to address the issue:
https://github.com/joyent/illumos-joyent/commit/192e1e6405f98e4b0a12f9488793c5dd000f3f7e

I would suggest including a comment where vbc_seg_max is set to note why that value is clamped as it is.

[richard_meric.id.au]

Added comment where vbc_seg_max is set to note why that value is clamped as it is (as recommended by pmooney_pfmooney.com).

[richard_meric.id.au]

In D18831#402095, @pmooney_pfmooney.com wrote:

This is effectively what we did in SmartOS/illumos to address the issue:
https://github.com/joyent/illumos-joyent/commit/192e1e6405f98e4b0a12f9488793c5dd000f3f7e

I would suggest including a comment where vbc_seg_max is set to note why that value is clamped as it is.

Thanks Patrick, I've included the comment from your patch as suggested.

The SmartOS patch was the source of my patch - I only raised it here so that it might find its way into the upstream code-base, as both other attempts at this seem to have stalled.
Definitely happy for the original author to claim it.

[araujo]

Hi Richard,

Do you mind share the tests you made??

Best,

[richard_meric.id.au]

In D18831#402109, @araujo wrote:

Hi Richard,

Do you mind share the tests you made??

Best,

I've currently got these changes running in a FreeNAS-11.2-RELEASE-U1 instance (FreeBSD 11.2 kernel) running a couple of Windows 2016 VMs plus one Centos 7 - all using virtio-blk disk devices without issue. I also have pfSense (FreeBSD 11.2) and OpenSuSE (Leap 42.3) VMs which I haven't tested yet - but can do if required.

[araujo]

In D18831#402110, @richard_meric.id.au wrote:

In D18831#402109, @araujo wrote:

Hi Richard,

Do you mind share the tests you made??

Best,

I've currently got these changes running in a FreeNAS-11.2-RELEASE-U1 instance (FreeBSD 11.2 kernel) running a couple of Windows 2016 VMs plus one Centos 7 - all using virtio-blk disk devices without issue. I also have pfSense (FreeBSD 11.2) and OpenSuSE (Leap 42.3) VMs which I haven't tested yet - but can do if required.

OK! Thanks for sharing!
I have a test-bed here where I will test this patch and make some disk stress testing during couple days.

[araujo]

This comment was removed by araujo.

[araujo]

NOTE: I blocked this patch, till I finish my analysis, nothing wrong with that...

[richard_meric.id.au]

Not sure about the seg_max value (even in the current code).
Looking at the qemu implementation, I suspect it should be queue length -2, which I guess is what the patched code happens to achieve (based on the definition of VTBLK_RINGSZ), but it could be more explicit.

Only remaining thing is whether we should do something different for the ahci device, as it's currently getting a 128 element IOV, but really only needs 32 (33?). I'm not sure if it is a problem having it larger?

[jhb]

usr.sbin/bhyve/block_if.c

65 ↗

(On Diff #52820)

Should this be using BLOCKIF_IOV_MAX going forward, or maybe something like MAX(64, BLOCKIF_IOV_MAX)?

[jhb]

• Added a new constant and a static assertion and some more comments to reduce magic numbers a bit.

[rgrimes]

This makes it much clearer to me, thanks jhb