Page MenuHomeFreeBSD

[PPC64] Implement CAS
ClosedPublic

Authored by luporl on Jul 1 2019, 5:57 PM.

Details

Summary

Guest PPC OSs running under a hypervisor may communicate the features they support, in order for the hypervisor to expose a virtualized machine in the way the client (guest OS) expects (see LoPAPR 1.1 - B.6.2.3).

This is done by calling the "/ibm,client-architecture-support" (CAS) method, informing supported features in option vectors.
Until now, FreeBSD wasn't using CAS, but instead relied on hypervisor/QEMU's defaults.

The problem is that, without CAS, it is very inconvenient to run POWER9 VMs on a POWER9 host running with radix enabled.
This happens because, in this case, the QEMU default is to present the guest OS a dual MMU (HPT/RPT), instead of presenting a regular HPT MMU, as FreeBSD expects, resulting in an early panic. The known workarounds required either changing the host to disable radix or passing a flag to QEMU to run in a POWER8 compatible mode.

With CAS, FreeBSD is now able to communicate that it wants an HPT MMU, independent of the host setup, which now makes FreeBSD work on POWER9/pseries, with KVM enabled and without hugepages (support added in a previous commit).

As CAS is invoked through OpenFirmware's call-method interface, it needs to be performed early, when OpenFirmware is still operational. Besides, now that FDT is the default way to inspect the device tree on PPC, OFW call-method feature will be unavailable by default, when control is passed to the kernel. Because of this, the call to CAS is being performed at the loader, instead of at the kernel.

Finally, to avoid regressions with old platforms, this change uses CAS only on POWER8/POWER9.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

luporl created this revision.Jul 1 2019, 5:57 PM
jhibbits added inline comments.Jul 9 2019, 9:33 PM
stand/powerpc/ofw/cas.c
199 ↗(On Diff #59271)

Would this prevent loader from being used on powernv? I know currently we don't boot with loader on POWER8 and POWER9, but that may change in the future.

luporl added inline comments.Jul 10 2019, 1:42 PM
stand/powerpc/ofw/cas.c
199 ↗(On Diff #59271)

Yes, it would.
What if I add a check for PSL_HV bit and skip CAS if it is set? Kernel code seems to use this check to skip code that is specific to virtualized environments.

jhibbits added inline comments.Jul 11 2019, 4:21 PM
stand/powerpc/ofw/cas.c
199 ↗(On Diff #59271)

That should work, I think.

luporl updated this revision to Diff 59848.Jul 17 2019, 6:58 PM

Skip CAS if PSL_HV bit is set on MSR

This avoids limiting loader to pseries systems only

luporl updated this revision to Diff 59849.Jul 17 2019, 7:01 PM
  • Fix inverted tab/newline in asm
luporl marked 2 inline comments as done.Jul 17 2019, 7:03 PM
luporl added inline comments.
stand/powerpc/ofw/cas.c
199 ↗(On Diff #59271)

HV bit test added. I've confirmed it works on pseries.

luporl updated this revision to Diff 59854.Jul 17 2019, 7:15 PM
luporl marked an inline comment as done.
  • [PPC64] Implement CAS
  • Skip CAS if PSL_HV bit is set on MSR

I've posted a previous diff over an already patched file, that temporarily screwed the patch/history.

It is fixed now, sorry.

jhibbits added inline comments.Jul 30 2019, 1:36 PM
stand/powerpc/ofw/Makefile
42 ↗(On Diff #59854)

do we need debugging enabled unconditionally?

stand/powerpc/ofw/cas.c
181 ↗(On Diff #59854)

Can you use machine/psl.h and machine/cpufunc.h? Then you can use 'uint64_t hv = mfmsrd() & PSL_SF; return (hv != 0);'

luporl marked an inline comment as done.Jul 30 2019, 1:57 PM
luporl added inline comments.
stand/powerpc/ofw/Makefile
42 ↗(On Diff #59854)

Yes, we do. Other parts of stand already do this.

The loader is stripped before being installed, so that all debug info is removed.

But without this, loader.full was inconsistent during debugging: some parts of it had debugging symbols, while others had not.

stand/powerpc/ofw/cas.c
181 ↗(On Diff #59854)

Ok, I'll give it a try and see if works.

luporl marked 2 inline comments as done.Jul 30 2019, 4:40 PM
luporl added inline comments.
stand/powerpc/ofw/cas.c
181 ↗(On Diff #59854)

It doesn't work, as doing so raises a couple of issues.

The first issue is that stand is 32-bit, so __powerpc64__ is not defined and would need to be defined manually to get the right contents from psl.h and cpufunc.h.

The second issue is that the needed contents are protected by an "#ifdef _KERNEL", needing to define it manually to make it work.

The third issue, that I haven't investigate deeper, is that register_t ends up being defined as a 32-bit integer, which truncates the value returned by mfmsr(), resulting in wrong code that will test a bit that will always be 0.

So in summary it seems that, at least currently, it is not possible to reuse cpufunc.h and psl.h in stand.

jhibbits accepted this revision.Jul 31 2019, 6:48 PM

Thanks for checking. Looks good. I assume you tested on bare metal, too?

This revision is now accepted and ready to land.Jul 31 2019, 6:48 PM

Thanks for checking. Looks good. I assume you tested on bare metal, too?

Actually, I've tested on VMs only, as this changes loader only, that is not used on POWER8 and above.
Do you want to test this on a G5? Or is there a way to test loader under PetitBoot?

Thanks for checking. Looks good. I assume you tested on bare metal, too?

Actually, I've tested on VMs only, as this changes loader only, that is not used on POWER8 and above.
Do you want to test this on a G5? Or is there a way to test loader under PetitBoot?

You're right. Nevermind. Good to go.

This revision was automatically updated to reflect the committed changes.