Offer VMX to guests when the hw.vmm.vmx.nested tunable is set, so that a guest (L1) can run virtual machines of its own (L2). This is the smallest set of features that runs an L2. It is meant as the base that the rest (EPT for L1, VPID, VMCS shadowing, a switch per VM) is added to in separate changes.
- L1 sees CPUID.1:ECX.VMX, IA32_FEATURE_CONTROL and the VMX capability MSRs. It is not offered EPT, VPID, unrestricted guest, APIC virtualization, the TPR shadow, VMCS shadowing, I/O or MSR bitmaps, MSR load/store areas or the preemption timer, so that it has to run L2 on shadow page tables, and it has to be a 64-bit hypervisor. The VM exits that the host needs L2 to have in any case are reported as controls that L1 must set: every exit that L1 gets is one that it asked for.
- VMXON, VMXOFF, VMCLEAR, VMPTRLD, VMPTRST, VMREAD and VMWRITE are emulated on L1's VMCS ("vmcs12"), a page of its memory that is cached in the vCPU while it is the current one.
- VMLAUNCH and VMRESUME check vmcs12 against the capability MSRs and merge it into a second VMCS of the vCPU ("vmcs02"), which vmx_run() enters instead of L1's. vmcs02 uses the VM's EPT unchanged, since L2's page tables hold L1-physical addresses. Nothing but L2's state, the event to inject and the controls that L1 was offered is taken from vmcs12.
- A VM exit of L2 that is the host's own (a host interrupt or NMI, an EPT violation in L1's memory) is handled in vmx_run_nested(). The others are reflected to L1, which continues with the state of vmcs12's host-state area. An interrupt or NMI for L1 that arrives while L2 runs is an external-interrupt or NMI exit of L2.
- VM_EXITCODE_VMINSN is passed to the new vmm_ops method vminsn with the vCPU frozen, because the emulation reads and writes the guest's memory. For AMD it injects #UD as before.
With the tunable set, APIC virtualization is not used for any guest, and a vCPU that is in VMX operation cannot be snapshotted.
Where this code comes from
keelOS (keelos.dev), a hypervisor appliance built on FreeBSD and bhyve, has had nested VMX since October 2026, including EPT for L1, VMCS shadowing and Hyper-V as L1. That implementation cannot be moved to FreeBSD as it is: keelOS's vmm.ko does not own VMX root operation. A small hypervisor underneath the FreeBSD kernel holds the VMCSs, and vmm.ko reaches them through hypercalls.
This change is a new implementation of the first stage of that work for stock vmm.ko, where vmcs02 is an ordinary second VMCS of the vCPU. The instruction emulation, the checks, the set of capability MSRs and the rules for reflecting VM exits were carried over from keelOS; the VMCS handling, the run loop for L2 and the vmm_ops hook were written for this change.
References: the emulation follows the Intel SDM, Vol. 3C (VMX instructions, VM entries and VM exits). Linux's KVM was read for the overall scheme (the names vmcs12 and vmcs02 come from it) and to learn what it requires from the CPU when it runs as L1. No code was taken from KVM or from any other hypervisor.
AI assistance: this change was developed with an AI assistant (Anthropic's Claude) under my direction, as keelOS itself is. The assistant wrote the code and this description, working from the keelOS implementation and the SDM. The tests in the test plan were run on real hardware, and the emulation was compared with the SDM as described there.
Signed-off-by: Wanpeng Qian <wanpengqian@gmail.com>
Sponsored by: keelos.dev