allocate pcpu and dpcpu from correct domain

[kib]

For dpcpu the change should be fine.

For pcpu, I do not think so, it breaks counters(9). Counters work by taking fixed offset against the pcpu base, and special counters pcpu zone allocator arranges for it. Look at the assembly in the MD headers for x86 counter.h to see the magic.
Also, the change lacks i386 counterpart, but it does not matter until the issue above is not resolved.

sys/amd64/amd64/mp_machdep.c

413

Continuation line should use 4-spaces indent.

[kib]

In fact, this pcpu change should break more. I am not sure about the pmap PCID initialization, it depends on the order, but for instance bhyve should be broken as well.

Grep for __pcpu to see.

[mmacy]

In D16158#342678, @kib wrote:

For dpcpu the change should be fine.

For pcpu, I do not think so, it breaks counters(9). Counters work by taking fixed offset against the pcpu base, and special counters pcpu zone allocator arranges for it. Look at the assembly in the MD headers for x86 counter.h to see the magic.
Also, the change lacks i386 counterpart, but it does not matter until the issue above is not resolved.

counters use their own zone the sizeof(struct pcpu) was an arbitrary offset for spacing. They are now UMA_PCPU_ALLOC_SIZE (PAGE_SIZE) apart so that counters can be allocated from the right domain's memory.

I can't comment on PCID or bhyve yet.

[kib]

In D16158#342699, @mmacy wrote:

In D16158#342678, @kib wrote:

For dpcpu the change should be fine.

For pcpu, I do not think so, it breaks counters(9). Counters work by taking fixed offset against the pcpu base, and special counters pcpu zone allocator arranges for it. Look at the assembly in the MD headers for x86 counter.h to see the magic.
Also, the change lacks i386 counterpart, but it does not matter until the issue above is not resolved.

counters use their own zone the sizeof(struct pcpu) was an arbitrary offset for spacing. They are now UMA_PCPU_ALLOC_SIZE (PAGE_SIZE) apart so that counters can be allocated from the right domain's memory.

No. Let me formulate it this way: &counter[cpuN] - &counter[cpu0] == &pcpuN - &pcpu0 is the invariant which makes the counters(9) asm work on x86.

I can't comment on PCID or bhyve yet.

[mmacy]

In D16158#342700, @kib wrote:

In D16158#342699, @mmacy wrote:

In D16158#342678, @kib wrote:

For dpcpu the change should be fine.

For pcpu, I do not think so, it breaks counters(9). Counters work by taking fixed offset against the pcpu base, and special counters pcpu zone allocator arranges for it. Look at the assembly in the MD headers for x86 counter.h to see the magic.
Also, the change lacks i386 counterpart, but it does not matter until the issue above is not resolved.

counters use their own zone the sizeof(struct pcpu) was an arbitrary offset for spacing. They are now UMA_PCPU_ALLOC_SIZE (PAGE_SIZE) apart so that counters can be allocated from the right domain's memory.

No. Let me formulate it this way: &counter[cpuN] - &counter[cpu0] == &pcpuN - &pcpu0 is the invariant which makes the counters(9) asm work on x86.

I see now. Thanks. It's unfortunate that that didn't come up on D15933 as it would seem that I've now broken counters on amd64. Request permission to make the code use atomics until I can put in a proper fix. (It's 02:47 and I was about to go to bed)

[kib]

In D16158#342703, @mmacy wrote:

I see now. Thanks. It's unfortunate that that didn't come up on D15933 as it would seem that I've now broken counters on amd64. Request permission to make the code use atomics until I can put in a proper fix. (It's 02:47 and I was about to go to bed)

I would revert r336020, but it is not obvious to me that this is needed. Can you elaborate why D15933 broke it ? If the issue is in the stride size, then struct pcpu should be extended to have the size of the page.

[mjg]

Is this obsolete (or about to be) due to recent work?