Update

Update

In D27855#622565, @jrtc27 wrote:

In D27855#622543, @bryanv wrote:

In D27855#622509, @jrtc27 wrote:

In D27855#622508, @bryanv wrote:

In D27855#622493, @jrtc27 wrote:

Everything those commits did is necessary for V1. It's not sufficient per the spec, granted, but having the commits to support _some_ V1 implementations is strictly better than supporting _none_. If you have commits to introduce the missing parts of the V1 feature negotiation sequence then add them, but don't revert what's already there only to reintroduce it later. Otherwise you will break TinyEMU and the FPGA-based system we have built atop its VirtIO implementation.

It is so wrong and incomplete that I have zero interest in trying to sort out the merge conflicts in the patches series I have for V1 support. This is so far away from _some_ V1 support that it is effectively _none_. Fix TinyEMU: it would appear it otherwise supports legacy but has a bug that requires VIRTIO_F_VERSION_1 to be negotiated.

That is grossly incorrect. There are a couple of deficiencies, and as I just said, everything I did is required for V1.

No, it is not just "a couple". I would advise you to read the VirtIO V1 spec. Or look at D27856 that starts to add V1 support. The OASIS TC did not spend years to develop a V1 specification that just ... negotiates VIRTIO_F_VERSION_1 on top of legacy VirtIO. Doing so here is so far from any V1 support that it is wrong pointless.

I have read the spec. MMIO v1 is not all that different from MMIO v0.95; PCI changed a fair amount, but not MMIO. And a lot of v1 is just optional additional flexibility/features that FreeBSD does not try to negotiate, so are irrelevant. As for the changes to the configuration registers themselves, those have already been implemented. The only thing missing is proper endianness handling for big-endian systems, something which VirtIO has a poor track record of anyway, and is only needed if you have both a modern device and a big-endian guest, something which wouldn't have worked before anyway.

What I don't understand is why not just fix TinyEMU? It must already basically support legacy because that is ultimately what ends up being used here, but it would appear to have a bug that requires VIRTIO_F_VERSION_1 to be negotiated. This is NOT "_some_ V1 implementation"; it is V-TinyEMU, with no good reason.

It's not legacy; see below.

I pointed out to you privately at the time that the change to vtnet_rx_header broke other assumptions but I never got a response, so there is little that is salvageable from these commits.

There was a regression that I fixed very quickly for one specific case that bhyve hit. I started to draft a response to your private email but quite frankly it was extremely rude and condescending so I gave up and decided it was best to not engage in such conversation. And saying it's "wrong and incomplete", "effectively _none_" and "there is little that is salvageable" just continues in that manner and I am not enthused about working with you to improve VirtIO support if that's how you want to treat my contributions.

My email was not about that regression. For the record, here is the email I sent you:

Please be sure to add me (bryanv@) on any VirtIO code reviews in the future. I'm working on rebasing some older work I have to add actual V1 support.

Note that you broke the previous assumption around the RX padding by adding 2 bytes to the structure. And for this change in general, we should not be advertising half-baked V1 support like this. This is not non-legacy support. At quick glance, this looks to be exploiting non spec compliant behavior in the TinyEMU.

Please try to minimize changes in this area because it just makes rebasing V1 support harder.

I'm not quite sure what part is "extremely rude and condescending".

Yes, "... you broke the previous assumption around the RX padding by adding 2 bytes": by adding virtio_net_hdr_mrg_rxbuf to the vtnet_rx_header WITHOUT adjusting VTNET_RX_HEADER_PAD, the IP header no longer lands on a word boundary. We all introduce breakages, but being told so isn't rude.

That's not a breakage though, that's just a performance regression, and can trivially be fixed by changing the 4 to a 2 (or, better, replacing it with 4 - (sizeof(union) % 4) so it's never wrong).

In D27855#622509, @jrtc27 wrote:

In D27855#622508, @bryanv wrote:

In D27855#622493, @jrtc27 wrote:

Everything those commits did is necessary for V1. It's not sufficient per the spec, granted, but having the commits to support _some_ V1 implementations is strictly better than supporting _none_. If you have commits to introduce the missing parts of the V1 feature negotiation sequence then add them, but don't revert what's already there only to reintroduce it later. Otherwise you will break TinyEMU and the FPGA-based system we have built atop its VirtIO implementation.

It is so wrong and incomplete that I have zero interest in trying to sort out the merge conflicts in the patches series I have for V1 support. This is so far away from _some_ V1 support that it is effectively _none_. Fix TinyEMU: it would appear it otherwise supports legacy but has a bug that requires VIRTIO_F_VERSION_1 to be negotiated.

That is grossly incorrect. There are a couple of deficiencies, and as I just said, everything I did is required for V1.

In D27855#622493, @jrtc27 wrote:

Everything those commits did is necessary for V1. It's not sufficient per the spec, granted, but having the commits to support _some_ V1 implementations is strictly better than supporting _none_. If you have commits to introduce the missing parts of the V1 feature negotiation sequence then add them, but don't revert what's already there only to reintroduce it later. Otherwise you will break TinyEMU and the FPGA-based system we have built atop its VirtIO implementation.

In D26933#618091, @jhb wrote:

In D26933#617195, @bryanv wrote:

This looks odd to me having to push this down into the driver. I don't really like how to code flows with the 'msix_load' variable. I'm not sure what your VMM here is, but I'd much rather only apply this to transitional or modern devices.

Where else would you put it? It's hard to do in the PCI bus itself because the resource is often used for other things in the driver and you can only bus_alloc_resource() "once" for a given BAR. Most hardware stores the table and PBA in fixed locations which don't require quite this amount of dynamic behavior. In terms of VirtIO drivers, this file seems like the right place to do it?

This looks odd to me having to push this down into the driver. I don't really like how to code flows with the 'msix_load' variable. I'm not sure what your VMM here is, but I'd much rather only apply this to transitional or modern devices.

I have added support for this feature (and more) in my VirtIO V1 branch that I plan on committing soon.

Sorry I wasn't clearer: use virtqueue_size() after the allocate to do the check. Do not add a new virtqueue_size bus interface.

I don't think this is the right approach. Don't add a new virtqueue_size bus interface. Use virtqueue_size() after the allocate VQ call. VirtIO V1 supports VQ resizing that I would like to support later so for cases like this we'd need additional checking after the allocate anyways.

And does it make sense to limit to sub-256kB ios when linux qemu allows 512kB?

I don't know of a reason, but I'm not familiar with this area. @bryanv?

Essentially, yeah: virtqueue_enqueue: vtscsi0 request - too many segments to enqueue: 65, 64/0

What is the panic message?

@cem LGTM

@cem LGTM

Other drivers following the callout pattern seem to be: bcm2835_rng.c, octeon_rnd.c, hifn7751.c, glxsb.c, tpm20.c although perhaps there is a desired functional difference that I am not aware of.

@markm: Can you explain the difference in the prior version of this driver that is callout based and this change? It seems there is at least one driver in the tree that uses the callout pattern.

It has been a long while since I wrote this driver but the callout pattern was certainly cribbed from another driver in the tree. Has this API changed over the years?

This is fine. I made a similar change in my VirtIO V1 branch (https://github.com/bryanv/freebsd/commit/a2903365a44b710929e52bf00b85a501856a7510) but I have not had the time to commit it yet.

Please reread my prior comment. This is not correct.

This is not correct. You are quoting from the V1 spec but the FreeBSD VirtIO drivers are written against the pre-V1 (0.9.5 aka legacy) which uses guest endian.

Can you update this diff with the full context? Thanks.

In D1435#343209, @mmacy wrote:

@bryanv What do you feel remains to be done?