Thanks, looking at this now.

Thanks kib@, I can commit this.

I'm happy with this change but need to wait for maintainer approval before I commit it

In D26035#578286, @jhb wrote:

In D26035#578062, @grehan wrote:

Given the config file syntax will likely change but the "-f" option will be around forever, can a magic number or first-line metadata comment be used to distinguish this key/value file format from a future version ?

I would be fine with using a more obscure option for the flat config file (if we did long opts it could be a --flat-config or some such without a short letter even, but we don't do any long opts so not quite as straightforward to pick an obscure one). I don't really expect the syntax of that to change, but I think having '-f' be used for a more friendly UCL syntax might be more useful and I would be happy to leave "-f" free for the UCL thing to use. The UCL thing might itself include a version number as a field for the UCL parser to use that I think would permit some level of compatibility in the future.

Given the config file syntax will likely change but the "-f" option will be around forever, can a magic number or first-line metadata comment be used to distinguish this key/value file format from a future version ?

In D25396#565538, @chuck wrote:

As an update, there is work underway to significantly enhance bhyve configuration. Once that lands, it should make many of the concerns here easier to address.

bhyve bits look fine.

happy for a MFC in say 2 weeks after commit to head?

Nice change. Don't forget to update the man page.

That all sounds reasonable: looks fine to me.

I don't have a problem with the serial, f/w rev or model number parts of this.

Relevant guest commits:
https://github.com/torvalds/linux/commit/2420a0b1798d7a78d1f9b395f09f3c80d92cc588
https://github.com/openbsd/src/commit/d72f70aa5ac4d8ade3f10d3366ab796d8ea2b9c1

Thanks for taking a look! Does this approach seem plausibly reasonable, outside of the implementation-detail type issues already raised?

Jason Tubnor has tested the MFC on 11-stable so I'll commit once acked by bz.

And this is indeed the AHCI controller. MSI is being used in the known-working 19.10, but there is no migration going on there. Disabling the MSI capability on the controller and falling back to legacy interrupts allowed an 8-vCPU install to complete and also showed the interrupt being migrated to another vCPU during the install (from /proc/interrupts).

virtio uses MSI-x on Linux so the MSI issue isn't directly related to that.

Rather than locking, I'm thinking this might be the lack of PBA emulation for MSI-x
(which in turn will require locking, though slightly different)

I don't think the tread/twrite routines need any locking: the updates are memory operations. Modifying an MSIx table is always going to take multiple writes so the lock can't protect the update.

In D24463#539490, @cem wrote:

I see; it seems like there is some additional instruction emulation specific to APIC accesses in sys/amd64/vmm/io, and those ranges aren't exposed to userspace's MMIO.

In D24463#539489, @cem wrote:

I'm not seeing how we ever register a MMIO region for LAPIC_PADDR or IOAPIC_PADDR. Do we?

My understanding is that gpa is precomputed for me and I don’t need to add displacement or compute it from regs. Is that right?

I'll have a closer look at the emulation. Might even break out Intel XED for some comparison testing.

Debugging the APIC is a slog. Start with KTR compiled in an single vCPU guests. See what accesses are being done with BEXTR and what the device emulation is returning. Use FreeBSD as a guest and ddb to put in guest breakpoints to allow the ktr log to be examined. You may have to spend a lot more time with SDM Vol3 Ch10 than you would care for.

Nice change. GCE has moved instruction emulation out of kernel KVM, and it would be interesting for bhyve to also have that ability: this is a step towards that.

Clang 10 -march=native kernels on znver1 emit BEXTR for APIC reads

There is a slightly hackier version of this and a harness at http://www.freebsd.org/~grehan/itest.tgz

That problem has since been fixed, and I've found NVMe to work well now.

(I'm not sure if the nvme spec requires the bar to be 64-bit capable.)

In D24107#532678, @bcran wrote:

In D24107#532656, @grehan wrote:

There is no need to error out on overflow here. The type17 isn't used by most o/s's for determining how much memory is available: there is e820, the EFI memory map, etc etc. Perhaps print an error in bhyve, buit there doesn't seem any point in erroring out.

Okay. Also, I don't think we're going to have guests configured with ~2PB memory any time in the near future :)

In D24107#532572, @bcran wrote:

• Rework type17 generation and error out on overflow.

add GUID value for discovering SMBIOS tables in UEFI, which iirc would help our UEFI situation

Looks good.

So it will be a choice of a UEFI firmware that does VNC and passthru but no NVMe or one
that does VNC and NVMe, that will be confusing.

I'm assuming this is an illumos specific issue, but it might be worth to test this on FreeBSD too.

Yamagi has mentioned some small optimizations that could be added to this (e.g. saving the TPR in the vcpu struct and only writing to the VMCS if there has been a change) but this can be done in future commits.

Looks good.

Is this good to go?

The alternative solution (the same used by QEMU) is to read the packet in a local bounce buffer (64KB), so that we figure out its actual length and then we call vq_getchain() the corrent number of times (e.g. just once for 1500 bytes packets). The cost for this solution is an additional packet copy, which is not good imho.

I'd suggest going even further by not making it an option and fixing it as always on. There doesn't seem to be any reason to have it disabled.

I think this change breaks x2apic mode (and Yamagi might have a patch for that) so I'd hold off committing for a bit.

I don't understand how to parse that sentence (or how it would work). Is this just a function of our EFI stack being a borrowed blob from TianoCore EDK II?

Yeah, I first learned of this horror when an ACPICA update broke Bhyve because the syntax the compiler recognized for FADT or similar changed. It's... not as crazy as it might seem. It might be preferable if the compiler was linked in instead of system()'d, but whatever.

A note for future work: with Windows, the ACPI tables are generated from within UEFI. You would need either a tag on the UUID so EFI could search for it (like it does with the SMBIOS tables), or use the fw i/f to allow UEFI to query the address of the UUID. I can help with that.

A worthwhile change.