Good to go with the name change.

I checked with Leon and he thought this was fine. Will submit pending the copyright issue.

Update style as per jhb.

Yes, this code was submitted by Matt M.

bhyve passthru is currently restricted to the subset of devices that can support MSI/MSI-x so this doesn't come into play for older PCI devices.

This looks fine, though I'm interested in the circumstances that this can happen - can module-load pre-empt an existing kernel thread that is in the midst of invltlb_glob() ?

I think this should be fine. Can always be tweaked later.

Fine by me. Looking forward to using this :)

Anything available I could use to test this on AMD ?

Adding Tycho to this.

Needs capsicum patches or it won't work (testing was done with Capsicum disabled).

So a comment on this: in general, api's are not added to FreeBSD that don't have base-system clients that use them, or for a good reason. I think this falls into the latter but I'd like to cut it down a bit.

• can the get_unrestricted_guest() routine be removed ? There is an error return on the set, which seems like it can be used to determine if unrestricted mode is not available (e.g. that's how bhyve uses the ioctl).
• is there a need for vcpu_reset() ? The BSP should be initialized to power-on state.. That could be a bug in bhyve and better to have it fixed there.

so I do not see why running a bunch of tests would yield.

You might have to explain further why you think you are exempt from having to demonstrate testing with this.

I'd like to see some guest test results (FreeBSD, Linux, Windows) for varying numbers of these, in particular, with sockets > 1, and comparing what Qemu shows for the same configuration.

This is an excellent change :)

You may be able to get away with just the "set" call. Given that all h/w that is supported by bhyve, with the exception of only Nehalem systems, support this, you may want to use just the error from the set call to handle this low-probability error.

Thanks for getting back to this Rod.

Only minor issue I had was the trailing underscores in the inline routine name: wasn't too sure if that was idiomatic FreeBSD. Maybe "_int" ?

Update from Leon:

Closing this revision - moving to D14022

Yep, that's correct. The code in the new review is based on this code so it's really a continuation of the effort.

Chuck - the review for this is now in D14022, where I believe your comments have already been addressed.

Some minor fixes from Leon, and an initial man page update.

I don't think I'll be able to get the CR8 stuff done as quick as I'd hoped, so I think it's fine to go ahead with this as-is. In addition, both Andriy and Anish have reported that Windows guests come up fine so the issue may not be as big a deal as I'd previously thought.

I'll pull in the man page diff (maybe modified). Leon also sent me an update so I'll roll both in.

The updated version of this code is at https://reviews.freebsd.org/D14022

Leon Dang has also been working on NVMe emulation, and his version works with Linux, Windows and UEFI boot. I'll post that code for review since it is a bit more recent and tested. In the meantime, that version can be seen at www.freebsd.org/~grehan/pci_nvme.c

@avg I will look at and test D13828.

Tested the SVM codepath with a Win10 guest on an AMD Sempron APU.

Add bhyve review group.

Dropping kern.hz to 50 on the Sempron seemed to show the problem a bit more: the 2nd phase of a Win10 install wasn't able to complete. With the diff, the install was able to complete. Subjectively things seemed quicker, particularly the disk-writing part of the install.

Some more info on how timing-sensitive this is: the 10.3 install Ryzen insta-repro doesn't happen with a 4 vCPU guest on a 1.3Gzh Sempron 3850 APU, nor with 4/6/8 vCPU guests on an Opteron 6320 :( Also, Win10 Pro 2 vCPU guests install fine on both systems, where the Ryzen shows a lockup very quickly.

The VIRQ injection doesn't cover all cases - it misses out the modification of the TPR register via CR8.

CR8 write exits are required or the same situation will occur as with VIRQ - the update written to the V_TPR needs to be acted on immediately by the APIC emulation code since an interrupt that was blocked may need to be injected.

Good to know that it fixes the issue on the Phenom. My Ryzen 1700 is pre-July17 so can't necessarily be trusted.

Also, I think CR8 write exits are needed to force re-evaluation of the TPR for the software APIC model.

I didn't have success with this on my Ryzen 1700, but it's also possibly buggy :( I'll try with an Opteron tomorrow.

This looks fine. I'll test on AMD. Some comments:

Adding tychon@ as a subscriber: he's been doing some 9P work in recent times.

Fine with the minor additions.

Thanks for this patch.

The OR emulation has been checked in with r325261

The OR emulation has been checked in with r325261

Matt Macy has some additional reset code for non-FLR devices (in his case GPGPUs).

I disabled capsicum by commenting out the cap_enter() call, and with the above struct revert, was able to cat a .WAV file and also use mpg123 to play a long mp3 file on a FreeBSD 11.1/amd64 guest, with a cheap USB audio adapter for output. Volume control seemed to work fine with mpg123. There was a very small glitch/skip when the audio was first played (or on resume after pause in mpg123).

Looking at this...

There are a bunch of minor style(9) issues.

Thanks - this keeps the behviour in sync with head where the routines are hidden in caph helpers, so my opinion is it is best to keep the catpages call.

Some high-level comments:

I checked the numbers and they look good.

I'm looking at this now. I think there will need to be some changes in the mechanics for this, and also the abstraction. I'll write up something longer with more detail.

Shutdown hooks will make it some day...