arm64: VM/PMAP changes for CCA guest support

[sarah.walker2_arm.com]

Summary

When in a realm:

• Mappings with mode VM_MEMATTR_DEVICE and VM_MEMATTR_DEVICE_NP are unprotected
• Imported busdma buffers in protected memory are always bounced
• If EARLY_PRINTK is in use, the UART physical address must be in the unprotected address space

Sponsored by: Arm Ltd

[andrew]

sys/arm64/arm64/efirt_machdep.c

233

Needs to be on pa not page table entry

sys/arm64/arm64/pmap.c

4237

5454–5455

This should be applied to pa as it may be wrong under FEAT_LPA2

5774

Is this correct when entering a large page, e.g. when psind == 2 a level 2 block will be created

[sarah.walker2_arm.com]

Address review comments

[alc]

I recently updated my Microsoft Dev Kit, and because of this change I'm seeing:

bus_dmamem_alloc failed to align memory properly.

The warning stems from an allocation by the nvme driver:

KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x44
bounce_bus_dmamem_alloc() at bounce_bus_dmamem_alloc+0x258
nvme_ctrlr_start() at nvme_ctrlr_start+0xbc8
nvme_ctrlr_start_config_hook() at nvme_ctrlr_start_config_hook+0x5ec
run_interrupt_driven_config_hooks() at run_interrupt_driven_config_hooks+0x94
boot_run_interrupt_driven_config_hooks() at boot_run_interrupt_driven_config_hooks+0x30
mi_startup() at mi_startup+0x1ec
virtdone() at virtdone+0x70

An added printf reports:

vaddr: 0xffffa000847d3f40, paddr: 1047d3f40, alignment: 1000

In this case, the misalignment appears to be harmless because page size alignment isn't actually needed.

[sarah.walker2_arm.com]

In D56599#1364175, @alc wrote:

I recently updated my Microsoft Dev Kit, and because of this change I'm seeing:

bus_dmamem_alloc failed to align memory properly.

Could you test https://reviews.freebsd.org/D59476 please?