[WIP] bhyve: GPU Passthrough for AMD Ryzen V1000

[corvink]

Summary

Description:
Add support of GPU Passthrough for integrated graphics of AMD Ryzen V1000 APUs.
Note:
This patch is currently not working on any system.

Prerequisites:

• AMD Ryzen V1000 APU with Vega graphics

How to enable GPU Passthrough:
Use the new "apu-d" device for your graphics card.
Your bhyve config could look like:

bhyve -c 1 -m 4G -HwS \
  -s 0,hostbridge \
  -s 2,apu-d,4/0/0 \
  -s 5,ahci-hd,/root/fbsd/fbsd.img \
  -s 6,virtio-net,tap10 \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
  fbsd

Tested Scenarios:
CPU: AMD Ryzen Embedded V1807B

OS
Windows	Not Working
Debian	Not Working
FreeBSD	Not Working

Installation Steps:

1. Create a FreeBSD image
2. Apply this patch to your FreeBSD sources
3. Rebuild kernel and bhyve
4. Apply following patches of sysutils_uefi-edk2-bhyve_files_2019-07-19.tgz to /usr/ports/sysutils/uefi-edk2-bhyve-csm:
   • patch-IntelFrameWorkModulePkg_Csm_LegacyBiosDxe_LegacyBootSupport.c
   • patch-IntelFrameWorkModulePkg_Csm_LegacyBiosDxe_LegacyPci.c
5. Rebuild /usr/ports/sysutils/uefi-edk2-bhyve-csm
6. Start vm with following config:

bhyve -c 1 -m 4G -HwS \
  -s 0,hostbridge \
  -s 2,apu-d,4/0/0 \
  -s 5,ahci-hd,/root/fbsd/fbsd.img \
  -s 6,virtio-net,tap10 \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
  fbsd

7. Press a key to enter Boot Menu
8. Boot your FreeBSD image (Boot Manager -> EFI Hard Drive)
9. Build gpu-firmware-kmod, drm-current-kmod and drm-kmod from ports
10. load amdgpu
    • kldload amdgpu

Known Limitations:

• Detection of VBIOS address only uses VFCT table. drm-kmod/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c contains more options

What I've done so far:

1. Extract PCI ROM support from vga-bios patches
2. Add an ioctl to get VBIOS from VFCT
3. Add an apu-d device
   • It's a passthru device
   • It maps VBIOS into PCI ROM
   • Passthrough VGA-Ports and VGA-Memory
4. Add a dummy pci device to allow shadowing of OpROM

Known Issues:

1. Load of amdgpu driver fails with following message:

<6>[drm] Found VCN firmware Version ENC: 1.9 DEC: 1 VEP: 0 Revision: 28
<6>[drm] PSP loading VCN firmware
<6>[drm] reserve 0x400000 from 0xf43f800000 for PSP TMR
<4>[drm] psp command failed and response status is (0x0)
[drm ERROR :psp_hw_start] PSP load asd failed!
[drm ERROR :psp_hw_init] PSP firmware loading failed
[drm ERROR :amdgpu_device_fw_loading] hw_init of IP block <psp> failed -22
drmn0: amdgpu_device_ip_init failed
drmn0: Fatal error during GPU init
<6>[drm] amdgpu: finishing device.
<6>[drm] amdgpu: ttm finalized
Warning: can't remove non-dynamic nodes (dri)!
device_attach: drmn0 attach returned 22

[grehan]

Generic PCI ROM BIOS support in EFI might be a better solution for this (there is a version posted in the vga-bios patches a while back), with ROM extraction being a separate issue.

Does Linux support APU passthru ?

[corvink]

In D26506#592106, @grehan wrote:

Generic PCI ROM BIOS support in EFI might be a better solution for this (there is a version posted in the vga-bios patches a while back), with ROM extraction being a separate issue.

Could you give me some more information on where to find these patches and how to apply them?

Does Linux support APU passthru ?

APU passthru on linux doesn't work. drm reports that it can't locate a BIOS ROM.

[    1.743534] [drm] BIOS signature incorrect ff ff
[    1.743538] amdgpu 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    1.764021] [drm] BIOS signature incorrect ff ff
[    1.764075] [drm:amdgpu_get_bios [amdgpu]] *ERROR* Unable to locate a BIOS ROM
[    1.764946] amdgpu 0000:00:02.0: Fatal error during GPU init

FreeBSD has the same issue. For the default display device FreeBSD searches at VGA_PCI_BIOS_SHADOW_ADDR (0xC0000) for a BIOS ROM instead of reading it address from PCI Config Space. Uncommenting this code path in sys/dev/pci/vga_pci.c solves the issue. That's why it's neccessary to rebuild the kernel of the FreeBSD guest. I don't know yet whether it's an issue of my APU passthrough or bhyve-edk2.
Currently I didn't tested, if a patched linux kernel would work.

[grehan]

Could you give me some more information on where to find these patches and how to apply them?

The work is at https://www.gulbra.net/freebsd-bhyve/

You'd need to extract the PCI ROM BIOS support. I can help or even do that.

Does Linux support APU passthru ?

I should have been clearer here: do you know if a Linux host with KVM supports APU passthru to a guest ?

APU passthru on linux doesn't work. drm reports that it can't locate a BIOS ROM.

[    1.743534] [drm] BIOS signature incorrect ff ff
[    1.743538] amdgpu 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    1.764021] [drm] BIOS signature incorrect ff ff
[    1.764075] [drm:amdgpu_get_bios [amdgpu]] *ERROR* Unable to locate a BIOS ROM
[    1.764946] amdgpu 0000:00:02.0: Fatal error during GPU init

FreeBSD has the same issue. For the default display device FreeBSD searches at VGA_PCI_BIOS_SHADOW_ADDR (0xC0000) for a BIOS ROM instead of reading it address from PCI Config Space. Uncommenting this code path in sys/dev/pci/vga_pci.c solves the issue. That's why it's neccessary to rebuild the kernel of the FreeBSD guest. I don't know yet whether it's an issue of my APU passthrough or bhyve-edk2.
Currently I didn't tested, if a patched linux kernel would work.

bhyve currently has guest RAM at that location. However, that needs to be changed to support VGA, and that would also allow a ROM segment to be mapped there.

[corvink]

In D26506#594727, @grehan wrote:

The work is at https://www.gulbra.net/freebsd-bhyve/

You'd need to extract the PCI ROM BIOS support. I can help or even do that.

Thank you for your information. It looks doable for me. So, I'm going to do it on my own.

I should have been clearer here: do you know if a Linux host with KVM supports APU passthru to a guest ?

I don't know if linux supports APU passthru and haven't tried it. Additionally, I can't find any information on that.

[corvink]

This patch adds a ROM emulation to bhyve. This was added by D33129. So, this patch isn't required any more.