Our build flags require all externally visible function definitions to be preceded by a declaration (-W-missing-prototypes).

I removed the declaration from bhyverun.h, so both .c files which refer to vmexit_task_switch() need a declaration. I think bhyverun.h is probably not the right place, but I'm also not sure yet how to deal with vmexit handlers: amd64 and arm64 have mostly different VM_EXITCODE_* definitions. So, I just put the declaration here for now.

Maybe the better approach is to replace the handler table with a linker set. That is, for each VM_EXITCODE_* handler, write:

static int
vmexit_task_switch(...)
{
}
VM_EXIT_HANDLER_DEFINE(VM_EXITCODE_TASK_SWITCH, vmexit_task_switch);

Then bhyve can build a lookup table before entering the main loop and use that instead of handler. What do you think?

It should be possible to create a symlink at build time, sure.

My suggestion in the review comments is to have separate bhyverun.c files for amd64 and arm64. In this scheme, the ifdef will be removed, so it is here only temporarily. In general I think that machine-dependent header files should only be included by machine-dependent .c files, so hopefully a machine symlink won't be necessary.

What about:

 --- bhyverun.h
extern vmexit_handler_t *handler;

int vmexit_machine_independent(struct vmexit *);

 --- bhyverun.c
vm_loop()
{
  vmexit = vm_run();
  if (vmexit->code < VM_EXITCODE_MAX && handler[vmexit->code] != NULL)
    handler[vmexit->code](vmexit);
}

 --- amd64/vmexit.c
static int
vmexit_machine_dependent(struct vmexit *vme)
{
  ...
}

struct vmexit_handler_t handler[VM_EXITCODE_MAX] = {
  [VM_EXITCODE_MACHINE_INDEPENDENT] = vmexit_machine_independent,
  [VM_EXITCODE_MACHINE_DEPENDENT] = vmexit_machine_dependent,
};

 --- arm64/vmexit.c
struct vmexit_handler_t handler[VM_EXITCODE_MAX] = ...;

Sound good.

This makes sense, I'll give it a try.

I think you might want to consider amd64/Makefile.inc instead which matches what we do in libc, etc. Just need to .include ${MACHINE_CPUARCH}/Makefile.inc and move the amd64-specific bits there?

I like this structure, though in the commit that adds that I think the prototype for this function is still duplicated? Long term I think you want an amd64/internal.h or some such that amd64/vmexit.c would be able to include that would have this prototype? (Maybe call it amd64/bhyve_md.h or some such to match some other places in the tree like libthr?)

The other long term strategy is to add CURDIR/MACHINE_CPUARCH to the -I path in CFLAGS if we find we need it, but I think having the #ifdef temporarily is fine.

Nit and only a suggestion: The diff in your next commit is a bit more readable by using

SCRS+= \
        fwctl.c \
        task_switch.c

That seems cleaner, thanks.

Yes, I haven't yet put the prototype into a header, mostly since I don't know what else might need to go in there yet. bhyve_md.h might be reasonable, yeah.

Yeah, I think I'd prefer to use explicit ifdefs as breadcrumbs for some later reshuffling and reorganization.

Yes, this happens in D40554, I just failed to update the makefiles in that patch.