Page MenuHomeFreeBSD

[WIP] bhyve: GPU Passthrough for AMD Ryzen V1000
Needs ReviewPublic

Authored by c.koehne_beckhoff.com on Sep 21 2020, 11:48 AM.

Details

Reviewers
None
Group Reviewers
bhyve
Summary

Description:
Add support of GPU Passthrough for integrated graphics of AMD Ryzen V1000 APUs.
Note:
This patch is currently not working on any system.

Prerequisites:

  • AMD Ryzen V1000 APU with Vega graphics

How to enable GPU Passthrough:
Use the new "apu-d" device for your graphics card.
Your bhyve config could look like:

bhyve -c 1 -m 4G -HwS \
  -s 0,hostbridge \
  -s 2,apu-d,4/0/0 \
  -s 5,ahci-hd,/root/fbsd/fbsd.img \
  -s 6,virtio-net,tap10 \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
  fbsd

Tested Scenarios:
CPU: AMD Ryzen Embedded V1807B

OS
WindowsNot Working
DebianNot Working
FreeBSDNot Working

Installation Steps:

  1. Create a FreeBSD image
  2. Apply this patch to your FreeBSD sources
  3. Rebuild kernel and bhyve
  4. Apply following patches of sysutils_uefi-edk2-bhyve_files_2019-07-19.tgz to /usr/ports/sysutils/uefi-edk2-bhyve-csm:
    • patch-IntelFrameWorkModulePkg_Csm_LegacyBiosDxe_LegacyBootSupport.c
    • patch-IntelFrameWorkModulePkg_Csm_LegacyBiosDxe_LegacyPci.c
  5. Rebuild /usr/ports/sysutils/uefi-edk2-bhyve-csm
  6. Start vm with following config:
bhyve -c 1 -m 4G -HwS \
  -s 0,hostbridge \
  -s 2,apu-d,4/0/0 \
  -s 5,ahci-hd,/root/fbsd/fbsd.img \
  -s 6,virtio-net,tap10 \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd \
  fbsd
  1. Press a key to enter Boot Menu
  2. Boot your FreeBSD image (Boot Manager -> EFI Hard Drive)
  3. Build gpu-firmware-kmod, drm-current-kmod and drm-kmod from ports
  4. load amdgpu
    • kldload amdgpu

Known Limitations:

  • Detection of VBIOS address only uses VFCT table. drm-kmod/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c contains more options

What I've done so far:

  1. Extract PCI ROM support from vga-bios patches
  2. Add an ioctl to get VBIOS from VFCT
  3. Add an apu-d device
    • It's a passthru device
    • It maps VBIOS into PCI ROM
    • Passthrough VGA-Ports and VGA-Memory
  4. Add a dummy pci device to allow shadowing of OpROM

Known Issues:

  1. Load of amdgpu driver fails with following message:
<6>[drm] Found VCN firmware Version ENC: 1.9 DEC: 1 VEP: 0 Revision: 28
<6>[drm] PSP loading VCN firmware
<6>[drm] reserve 0x400000 from 0xf43f800000 for PSP TMR
<4>[drm] psp command failed and response status is (0x0)
[drm ERROR :psp_hw_start] PSP load asd failed!
[drm ERROR :psp_hw_init] PSP firmware loading failed
[drm ERROR :amdgpu_device_fw_loading] hw_init of IP block <psp> failed -22
drmn0: amdgpu_device_ip_init failed
drmn0: Fatal error during GPU init
<6>[drm] amdgpu: finishing device.
<6>[drm] amdgpu: ttm finalized
Warning: can't remove non-dynamic nodes (dri)!
device_attach: drmn0 attach returned 22

Diff Detail

Repository
rS FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

Generic PCI ROM BIOS support in EFI might be a better solution for this (there is a version posted in the vga-bios patches a while back), with ROM extraction being a separate issue.

Does Linux support APU passthru ?

Generic PCI ROM BIOS support in EFI might be a better solution for this (there is a version posted in the vga-bios patches a while back), with ROM extraction being a separate issue.

Could you give me some more information on where to find these patches and how to apply them?

Does Linux support APU passthru ?

APU passthru on linux doesn't work. drm reports that it can't locate a BIOS ROM.

[    1.743534] [drm] BIOS signature incorrect ff ff
[    1.743538] amdgpu 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    1.764021] [drm] BIOS signature incorrect ff ff
[    1.764075] [drm:amdgpu_get_bios [amdgpu]] *ERROR* Unable to locate a BIOS ROM
[    1.764946] amdgpu 0000:00:02.0: Fatal error during GPU init

FreeBSD has the same issue. For the default display device FreeBSD searches at VGA_PCI_BIOS_SHADOW_ADDR (0xC0000) for a BIOS ROM instead of reading it address from PCI Config Space. Uncommenting this code path in sys/dev/pci/vga_pci.c solves the issue. That's why it's neccessary to rebuild the kernel of the FreeBSD guest. I don't know yet whether it's an issue of my APU passthrough or bhyve-edk2.
Currently I didn't tested, if a patched linux kernel would work.

Could you give me some more information on where to find these patches and how to apply them?

The work is at https://www.gulbra.net/freebsd-bhyve/

You'd need to extract the PCI ROM BIOS support. I can help or even do that.

Does Linux support APU passthru ?

I should have been clearer here: do you know if a Linux host with KVM supports APU passthru to a guest ?

APU passthru on linux doesn't work. drm reports that it can't locate a BIOS ROM.

[    1.743534] [drm] BIOS signature incorrect ff ff
[    1.743538] amdgpu 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    1.764021] [drm] BIOS signature incorrect ff ff
[    1.764075] [drm:amdgpu_get_bios [amdgpu]] *ERROR* Unable to locate a BIOS ROM
[    1.764946] amdgpu 0000:00:02.0: Fatal error during GPU init

FreeBSD has the same issue. For the default display device FreeBSD searches at VGA_PCI_BIOS_SHADOW_ADDR (0xC0000) for a BIOS ROM instead of reading it address from PCI Config Space. Uncommenting this code path in sys/dev/pci/vga_pci.c solves the issue. That's why it's neccessary to rebuild the kernel of the FreeBSD guest. I don't know yet whether it's an issue of my APU passthrough or bhyve-edk2.
Currently I didn't tested, if a patched linux kernel would work.

bhyve currently has guest RAM at that location. However, that needs to be changed to support VGA, and that would also allow a ROM segment to be mapped there.

The work is at https://www.gulbra.net/freebsd-bhyve/

You'd need to extract the PCI ROM BIOS support. I can help or even do that.

Thank you for your information. It looks doable for me. So, I'm going to do it on my own.

I should have been clearer here: do you know if a Linux host with KVM supports APU passthru to a guest ?

I don't know if linux supports APU passthru and haven't tried it. Additionally, I can't find any information on that.