I suggest to split the initialization into per-cpu allocations for AP. The sys/i386/i386/mp_machdep.c:init_secondary() is the natural place to do the init, right after lidt().

Why CADDR1 can be non-zero there ?

I do not see why do we need struct sysmaps, after the migration to pcpu. Before, it was needed to use it for array definition, now this purpose is gone. I suggest to inline the members into PCPU_MD_FIELDS.

Please use explicit number for pad array size.

It would be non-zero for the BSP. This check won't be necessary if initialization is moved to init_secondary().

I'm fine with the idea of inlining, but I kept struct sysmaps around for 2 reasons:

--logical grouping of the related members in a single struct

--reduction in number of calls to PCPU_GET(), which looks like it's a bit more expensive than a simple struct dereference (e.g. it does some size checks and re-reads %fs on i386, it requires a cp15 access on arm). The sysmaps object can be obtained with a single PCPU_GET(), while use of inlined fields would require 3-5 such calls for a typical pmap copy operation.

I could still get the same reduction in PCPU_GET() calls using pcpu_find(cpuid) as is done in the rmlock implementation, but that seems more convoluted.

The only reason I didn't leave it an explicit number is that this change adds an opaque element (struct mtx) to the pcpu data. Is there any concern about changes to struct mtx breaking the compile-time assertion on sizeof(pcpu) being the denominator of PAGE_SIZE?

I like your idea to move this to init_secondary(), but I don't know if I can get away with it.
I see a panic when I try it out because the kva_alloc() path takes a critical section, while scheduler and thread state haven't been initialized for the AP yet. VM layer should already be initialized by BSP, so I don't think bumping virtual_avail as is done in the bootstrap path is possible.
Is there some other allocation strategy I can use?

That makes sense. I'll change it to a constant.

You are absolutely right. Yes, we cannot take any sleepable locks or normal mutexes until scheduler takes over the AP initialization. Sorry.

I think that this could be fixed by providing some place for APs to perform the potentially blocking initialization before running any other code. E.g. we might create a runnable highest priority thread bound to AP just for that. I will think about this.

Would it be worth the trouble to do that though? If we only need this for per-CPU mappings, then it seems easier to just use the SI_SUB_CPU sysinit as done here. Are there other pieces of the system that could benefit from such a stage?

My main motivation for that is there we get non obvious hidden dependency in the initialization order. With the efforts to start APs much earlier than it is done now (see EARLY_AP_STARTUP recent work), this IMO becomes fragile. After I wrote this paragraph, I started wondering should asserts like CADDR1 != 0, CADDR2 != 0 added to consumers of sysmaps.

As I said above, do not postpone the change waiting for the initialization rework.