Thanks for taking a look! Does this approach seem plausibly reasonable, outside of the implementation-detail type issues already raised?

Jason Tubnor has tested the MFC on 11-stable so I'll commit once acked by bz.

And this is indeed the AHCI controller. MSI is being used in the known-working 19.10, but there is no migration going on there. Disabling the MSI capability on the controller and falling back to legacy interrupts allowed an 8-vCPU install to complete and also showed the interrupt being migrated to another vCPU during the install (from /proc/interrupts).

virtio uses MSI-x on Linux so the MSI issue isn't directly related to that.

Rather than locking, I'm thinking this might be the lack of PBA emulation for MSI-x
(which in turn will require locking, though slightly different)

I don't think the tread/twrite routines need any locking: the updates are memory operations. Modifying an MSIx table is always going to take multiple writes so the lock can't protect the update.

In D24463#539490, @cem wrote:

I see; it seems like there is some additional instruction emulation specific to APIC accesses in sys/amd64/vmm/io, and those ranges aren't exposed to userspace's MMIO.

In D24463#539489, @cem wrote:

I'm not seeing how we ever register a MMIO region for LAPIC_PADDR or IOAPIC_PADDR. Do we?

My understanding is that gpa is precomputed for me and I don’t need to add displacement or compute it from regs. Is that right?

I'll have a closer look at the emulation. Might even break out Intel XED for some comparison testing.

Debugging the APIC is a slog. Start with KTR compiled in an single vCPU guests. See what accesses are being done with BEXTR and what the device emulation is returning. Use FreeBSD as a guest and ddb to put in guest breakpoints to allow the ktr log to be examined. You may have to spend a lot more time with SDM Vol3 Ch10 than you would care for.

Nice change. GCE has moved instruction emulation out of kernel KVM, and it would be interesting for bhyve to also have that ability: this is a step towards that.

Clang 10 -march=native kernels on znver1 emit BEXTR for APIC reads

There is a slightly hackier version of this and a harness at http://www.freebsd.org/~grehan/itest.tgz

That problem has since been fixed, and I've found NVMe to work well now.

(I'm not sure if the nvme spec requires the bar to be 64-bit capable.)

In D24107#532678, @bcran wrote:

In D24107#532656, @grehan wrote:

There is no need to error out on overflow here. The type17 isn't used by most o/s's for determining how much memory is available: there is e820, the EFI memory map, etc etc. Perhaps print an error in bhyve, buit there doesn't seem any point in erroring out.

Okay. Also, I don't think we're going to have guests configured with ~2PB memory any time in the near future :)

In D24107#532572, @bcran wrote:

• Rework type17 generation and error out on overflow.

add GUID value for discovering SMBIOS tables in UEFI, which iirc would help our UEFI situation

Looks good.

So it will be a choice of a UEFI firmware that does VNC and passthru but no NVMe or one
that does VNC and NVMe, that will be confusing.

I'm assuming this is an illumos specific issue, but it might be worth to test this on FreeBSD too.

Yamagi has mentioned some small optimizations that could be added to this (e.g. saving the TPR in the vcpu struct and only writing to the VMCS if there has been a change) but this can be done in future commits.

Looks good.

Is this good to go?

The alternative solution (the same used by QEMU) is to read the packet in a local bounce buffer (64KB), so that we figure out its actual length and then we call vq_getchain() the corrent number of times (e.g. just once for 1500 bytes packets). The cost for this solution is an additional packet copy, which is not good imho.

I'd suggest going even further by not making it an option and fixing it as always on. There doesn't seem to be any reason to have it disabled.

I think this change breaks x2apic mode (and Yamagi might have a patch for that) so I'd hold off committing for a bit.

I don't understand how to parse that sentence (or how it would work). Is this just a function of our EFI stack being a borrowed blob from TianoCore EDK II?

Yeah, I first learned of this horror when an ACPICA update broke Bhyve because the syntax the compiler recognized for FADT or similar changed. It's... not as crazy as it might seem. It might be preferable if the compiler was linked in instead of system()'d, but whatever.

A note for future work: with Windows, the ACPI tables are generated from within UEFI. You would need either a tag on the UUID so EFI could search for it (like it does with the SMBIOS tables), or use the fw i/f to allow UEFI to query the address of the UUID. I can help with that.

A worthwhile change.

Good to go with the name change.

I checked with Leon and he thought this was fine. Will submit pending the copyright issue.

Update style as per jhb.

Yes, this code was submitted by Matt M.

bhyve passthru is currently restricted to the subset of devices that can support MSI/MSI-x so this doesn't come into play for older PCI devices.

This looks fine, though I'm interested in the circumstances that this can happen - can module-load pre-empt an existing kernel thread that is in the midst of invltlb_glob() ?

I think this should be fine. Can always be tweaked later.

Fine by me. Looking forward to using this :)

Anything available I could use to test this on AMD ?

Adding Tycho to this.

Needs capsicum patches or it won't work (testing was done with Capsicum disabled).

So a comment on this: in general, api's are not added to FreeBSD that don't have base-system clients that use them, or for a good reason. I think this falls into the latter but I'd like to cut it down a bit.

• can the get_unrestricted_guest() routine be removed ? There is an error return on the set, which seems like it can be used to determine if unrestricted mode is not available (e.g. that's how bhyve uses the ioctl).
• is there a need for vcpu_reset() ? The BSP should be initialized to power-on state.. That could be a bug in bhyve and better to have it fixed there.

so I do not see why running a bunch of tests would yield.

You might have to explain further why you think you are exempt from having to demonstrate testing with this.

I'd like to see some guest test results (FreeBSD, Linux, Windows) for varying numbers of these, in particular, with sockets > 1, and comparing what Qemu shows for the same configuration.

This is an excellent change :)

You may be able to get away with just the "set" call. Given that all h/w that is supported by bhyve, with the exception of only Nehalem systems, support this, you may want to use just the error from the set call to handle this low-probability error.

Thanks for getting back to this Rod.

Only minor issue I had was the trailing underscores in the inline routine name: wasn't too sure if that was idiomatic FreeBSD. Maybe "_int" ?

Update from Leon:

Closing this revision - moving to D14022

Yep, that's correct. The code in the new review is based on this code so it's really a continuation of the effort.

Chuck - the review for this is now in D14022, where I believe your comments have already been addressed.

Some minor fixes from Leon, and an initial man page update.

I don't think I'll be able to get the CR8 stuff done as quick as I'd hoped, so I think it's fine to go ahead with this as-is. In addition, both Andriy and Anish have reported that Windows guests come up fine so the issue may not be as big a deal as I'd previously thought.

I'll pull in the man page diff (maybe modified). Leon also sent me an update so I'll roll both in.