The updated version of this code is at https://reviews.freebsd.org/D14022

Leon Dang has also been working on NVMe emulation, and his version works with Linux, Windows and UEFI boot. I'll post that code for review since it is a bit more recent and tested. In the meantime, that version can be seen at www.freebsd.org/~grehan/pci_nvme.c

@avg I will look at and test D13828.

Tested the SVM codepath with a Win10 guest on an AMD Sempron APU.

Add bhyve review group.

Dropping kern.hz to 50 on the Sempron seemed to show the problem a bit more: the 2nd phase of a Win10 install wasn't able to complete. With the diff, the install was able to complete. Subjectively things seemed quicker, particularly the disk-writing part of the install.

Some more info on how timing-sensitive this is: the 10.3 install Ryzen insta-repro doesn't happen with a 4 vCPU guest on a 1.3Gzh Sempron 3850 APU, nor with 4/6/8 vCPU guests on an Opteron 6320 :( Also, Win10 Pro 2 vCPU guests install fine on both systems, where the Ryzen shows a lockup very quickly.

The VIRQ injection doesn't cover all cases - it misses out the modification of the TPR register via CR8.

CR8 write exits are required or the same situation will occur as with VIRQ - the update written to the V_TPR needs to be acted on immediately by the APIC emulation code since an interrupt that was blocked may need to be injected.

Good to know that it fixes the issue on the Phenom. My Ryzen 1700 is pre-July17 so can't necessarily be trusted.

Also, I think CR8 write exits are needed to force re-evaluation of the TPR for the software APIC model.

I didn't have success with this on my Ryzen 1700, but it's also possibly buggy :( I'll try with an Opteron tomorrow.

This looks fine. I'll test on AMD. Some comments:

Adding tychon@ as a subscriber: he's been doing some 9P work in recent times.

Fine with the minor additions.

Thanks for this patch.

The OR emulation has been checked in with r325261

The OR emulation has been checked in with r325261

Matt Macy has some additional reset code for non-FLR devices (in his case GPGPUs).

I disabled capsicum by commenting out the cap_enter() call, and with the above struct revert, was able to cat a .WAV file and also use mpg123 to play a long mp3 file on a FreeBSD 11.1/amd64 guest, with a cheap USB audio adapter for output. Volume control seemed to work fine with mpg123. There was a very small glitch/skip when the audio was first played (or on resume after pause in mpg123).

Looking at this...

There are a bunch of minor style(9) issues.

Thanks - this keeps the behviour in sync with head where the routines are hidden in caph helpers, so my opinion is it is best to keep the catpages call.

Some high-level comments:

I checked the numbers and they look good.

I'm looking at this now. I think there will need to be some changes in the mechanics for this, and also the abstraction. I'll write up something longer with more detail.

Shutdown hooks will make it some day...

Tested fine on a Ryzen 1700, passing through an em device to an 11.1 guest, and doing lots of network file transfers.

Thanks for this work. Let me give it a quick test (later today) to verify.

(interested in adding bhyve support for SGX)

Please credit Fabian Freyer (from D7029) as the original submitter of this code when committing.

There was already a review for this in D9033 where I pointed out the initial fix by myself that bumped the number of descs, and the subsequent backout when it broke Linux guests :(

I emailed Marcelo some changes I had to this - it doesn't work as is.

Any chance this could be committed ?

In D10014#219157, @glebius wrote:

I missed one thing yesterday. The default for vga is "on", not "io". Lines 363-364 of pci_fbuf.c.

Thanks for the clarification on v2.

You'll have to use openat(), with the fd set up prior to cap_enter(). Here's a small example from theraven@freebsd.org, who I'm sure will be happy to provide further advice :) http://www.informit.com/articles/article.aspx?p=1924012&seqNum=2

Any thoughts on the interaction with capsicum ? Seems like you just need blanket access underneath a particular directory.

Fine to commit after the max fbuf res added.

Fine by me. I didn't check, but were all the comments in this applied to the new revision ?

The most recent spec (Rev 3.00—December 2016, http://support.amd.com/TechDocs/48882_IOMMU.pdf) seems to indicate that v1 is the base features, and new features are described as options, so not sure how to determine if it's v2. The IVRS table does have a v1/v2 format discriminator, though that isn't printed out in the current code.

Tested fine on a Ryzen 1700 + Asus X370 mobo, with a dual-port Intel 82571 card. Both ports passed through to a FreeBSD 12-CURRENT amd64 4 vCPU guest, with iperf3/netperf stress tests run.

Many thanks for doing this. I'll get some detailed comments shortly.

That was mentioned in earlier review comments. Maybe something like a kassert_decl attribute (expanding to unused if INVARIANTS not defined) ?

Thanks for the info. I'll see if I can move this forward under a different phab review.

I emailed on 7/16/2016 asking for help on running gcc and didn't get a response.

Thanks for keeping this going Allan.

emaste - any suggestions for a good forum to discuss this ? Unfortunately it straddles the capsicum@ and virtualization@ lists :(

robak - this is fine to commit once the last minor changes are done.

Into the final straight now: doing a visual inspection and picking up minor style issues.

Click on the 'Download raw diff' button on the right-hand side menu. Apply that diff to a FreeBSD source tree and rebuild.

Still some of the test matrix to cover. Hopefully shortly.