xen/arm64: add xen early init

[ehem_freebsd_m5p.com]

Summary

Add the earliest stage of Xen guest initialization. This checks for the
presence of Xen and sets flags to indicate the need to initialize other
pieces (or not).

This was originally implemented by Julien Grall in 2014. This is three
portions broken off of a much larger commit.

Submitted by: Elliott Mitchell <[e-mail]>
Original implementation: Julien Grall <[e-mail]>, 2014-01-13 17:40:58

[royger]

sys/arm64/arm64/machdep.c

116

I think this needs to be somewhere else.

1271

This also needs to be in a separate Xen specific file.

1311

I think this needs to be more agnostic, so that each hypervisor doesn't need to add it's own hook here to detect presence. You would need some kind of generic hypervisor_early_init helper that could iterate over a list of hooks to detect hypervisors.

[royger]

If you only need such early detection for the console support, maybe you could consider placing it in the console probe hook instead?

[andrew]

sys/arm64/arm64/machdep.c

1260

Which node has the xen,xen compatible property? If it's at a known location we can check for that directly.

1311

How early does this need to run? If possible it's better to move things after cninit, although this may not always be possible, e.g. for console init.

[ehem_freebsd_m5p.com]

Looks like time for more examination and thinking on these.

sys/arm64/arm64/machdep.c

1260

Hmm, I wonder if @julien_xen.org can comment on this. Perhaps when this was originally implemented the location was less certain. Looking at a handy example, appears it should be /hypervisor.

1271

In the original it was part of sys/arm64/xen/xen-dt.c where it pretty clearly didn't belong either. I thought this was a short enough section not to warrant a separate file. In that case though what comes to mind is one of sys/arm64/xen/xen-init.c, sys/xen/xen-init.c or sys/xen/init.c. Then also have xen_init() in that file.

[julien_xen.org]

sys/arm64/arm64/machdep.c

1260

That's right, the path will always be "/hypervisor". If it is Xen, the compatible string will contain "xen,xen".

[ehem_freebsd_m5p.com]

sys/arm64/arm64/machdep.c

116

I'm starting to suspect breaking D29874 off of D29875 was a mistake. Thing is most of the bits are shared, but some portions don't really fit together that well.

1311

How early does this need to run? If possible it's better to move things after cninit, although this may not always be possible, e.g. for console init.

That is an interesting question and I don't actually know the answer. I can readily believe this is a relic of when @julien_xen.org was originally trying to implement FreeBSD/ARM/Xen support. Thinking get the console first, thus trying to get hypervisor interaction here.

Now things are well past initial bring-up, it may be unnecessary to have this run so early. I'm wondering whether this can be run as part of SI_SUB_HYPERVISOR. Mainly it needs to be run before any other portion of Xen (notably the Xen console device).

[ehem_freebsd_m5p.com]

What do you think of the following: Add a int xen_early_probe(void); to machine/xen/xen-os.h. For x86 this would be a #define xen_early_probe() xen_domain() in sys/x86/include/xen/xen-os.h. On ARM "xen_early_init()" would be renamed "xen_early_probe()" and would return the probe result.

In the xencons_cnprobe() function of sys/dev/xen/console/xen_console.c, replace the call to xen_domain() with a call to xen_early_probe().

I believe the Xen console code is the earliest point where Xen's presence/absence needs to be known. As such, for ARM this is a handy hook point. I plan to test this hypothesis, but it has to wait for a build.

sys/arm64/arm64/machdep.c

1311

I think this needs to be more agnostic, so that each hypervisor doesn't need to add it's own hook here to detect presence. You would need some kind of generic hypervisor_early_init helper that could iterate over a list of hooks to detect hypervisors.

1311

How early does this need to run? If possible it's better to move things after cninit, although this may not always be possible, e.g. for console init.

1311

I think this needs to be more agnostic, so that each hypervisor doesn't need to add it's own hook here to detect presence. You would need some kind of generic hypervisor_early_init helper that could iterate over a list of hooks to detect hypervisors.

1311

How early does this need to run? If possible it's better to move things after cninit, although this may not always be possible, e.g. for console init.

Quite early. The Xen console device has an interesting interface which could be hazardous to probe if Xen is absent; though this could simply be the Xen console driver assumes if Xen is present, it can interface and if Xen is absent who knows what it could get.

I think this needs to be more agnostic, so that each hypervisor doesn't need to add it's own hook here to detect presence. You would need some kind of generic hypervisor_early_init helper that could iterate over a list of hooks to detect hypervisors.

I'm actually rather surprised there isn't one already. My most recent build failed rather early, but analyzing things drew my attention to one potential hook point.

[ehem_freebsd_m5p.com]

Hmm, the editing function certainly isn't perfect. I thought I had told it to throw away those sections. Even though I can remove the file comment, it doesn't appear the in-line bits can be removed.

[julien_xen.org]

sys/arm64/arm64/machdep.c

1311

That is an interesting question and I don't actually know the answer. I can readily believe this is a relic of when @julien_xen.org was originally trying to implement FreeBSD/ARM/Xen support. Thinking get the console first, thus trying to get hypervisor interaction here.

I wrote the code quite a while ago and can't remember the exact reason. However, for debugging purpose it can be really helpful to have a console (at least output) working very early.

[ehem_freebsd_m5p.com]

I think what is in D29874 is going to merge into D29875. I've got comments resolved in my tree. Could be a bit before I get things updated here though. This has turned back into a pile of misc patches and needs to get turned back into a patch.

sys/arm64/arm64/machdep.c

116

This has been merged into D28982 and proposed as sys/xen/common.c.

1260

This is fixed in my tree, issue is when does it get updated here?

1271

Unfortunately this has rejoined sys/dev/xen/devicetree/xen-dt.c. This isn't a great fit for there though.

1311

This needs to be run very early. Good news is it doesn't need to be run quite this early. Instead the approach I mentioned already has demonstrated viability.

Rename the function "xen_early_probe()" and provide a return similar to xen_domain(). Then replace the call to xen_domain() in xencons_cnprobe(). That is when the presence or absence of Xen needs to be known. For x86 simply have a #define xen_early_probe() xen_domain() in x86/include/xen/xen-os.h.

[ehem_freebsd_m5p.com]

Marking D29874 as abandoned as the hunks have been scattered. The initial probe code has moved back to the device-tree code with D29875. The shared_info portions have joined D28982.