xen/grant_table: adjust use of pmap_kenter() for aarch64

[ehem_freebsd_m5p.com]

Summary

A really *gross* solution, but something which works for now. Hopefully
the next person is better with the proper machine-independent in-kernel
APIs for FreeBSD.

[ehem_freebsd_m5p.com]

I would strongly prefer a proper architecture-independent implementation, but this takes care of the immediate issue.

[mhorne]

In D29305#656182, @ehem_freebsd_m5p.com wrote:

I would strongly prefer a proper architecture-independent implementation, but this takes care of the immediate issue.

You might look at pmap_qenter() for this. Rather than physical addresses, it expects an array of vm_page_t. These can be obtained from resume_frames using the PHYS_TO_VM_PAGE macro.

There is also a pmap_qenter(9) man page.

sys/dev/xen/grant_table/grant_table.c

536

__i386__ is preferred to __x86__ within FreeBSD.

I don't see a need to add __arm__ to this list until someone tries to support it fully.

[ehem_freebsd_m5p.com]

Yeah, I'd already come to that conclusion, but one can sometimes
accidentally grab the wrong define.

[ehem_freebsd_m5p.com]

That man page does provide pretty good information. Right now though interrupt issues are a higher priority (appears the kernel is presently hanging due to interrupts, this version of the grant table doesn't appear to cause panic).

sys/dev/xen/grant_table/grant_table.c

536

Former: fixed.

Latter: While my goal is getting aarch64 operational, it seems worthwhile to try an aarch32 build at some point. In particular since getting aarch64 operational is likely to fix 95% of the aarch32 issues, it seems likely to be worth a little bit of effort. The one challenge is I'm unsure where the shared bits would go. Right now it really looks like nearly all of the issues for aarch64 are also issues for armel/armhf/aarch32 and riscv.

[royger]

sys/dev/xen/grant_table/grant_table.c

544

This is not correct, you seem to assume start_idx will always be 0, but that's not the case, see gnttab_expand for example.

You would need to use something like:

pmap_kenter((vm_offset_t)shared + start_idx * PAGE_SIZE,
	    (end_idx - start_idx + 1) << PAGE_SHIFT, resume_frames + start_idx * PAGE_SIZE,
	    VM_MEMATTR_DEFAULT);

Or alternatively maybe:

#ifdef __aarch64__
#define pmap_kenter(l, p) pmap_kenter(l, PAGE_SIZE, p, VM_MEMATTR_DEFAULT);
#endif

[ehem_freebsd_m5p.com]

sys/dev/xen/grant_table/grant_table.c

544

Gah! So close, but missed the effect on the one argument. Yeah, that might explain the latest failure I've seen. I'll be close to the former.

[ehem_freebsd_m5p.com]

Behind on pushing this update. As that fix is kind of crucial I had it
locally almost instantly.

[ehem_freebsd_m5p.com]

Hopefully things are sufficiently addressed now. This has been tested on an actual build and seems to be functioning.

I would prefer the Real Fix of using pmap_qenter() or other architecture-independent functions, but right now my goal is simply trying to reach a shell.

[ehem_freebsd_m5p.com]

Adding the hunk which depends upon both the main portion and D29351.

[ehem_freebsd_m5p.com]

D29305 does a sufficient job to abandon D29305.