Differential D27546
Avoid vm_free_count() for physmem available before VM ACPI walk. Authored by dgmorris_earthlink.net on Dec 10 2020, 5:20 PM. Tags None Referenced Files
Subscribers
Details Derive physical memory available in cpu_startup() from the phys_avail On some x64 platforms, this results in major amounts of memory not being Booted on platform showing the symptom with a modified version containing both versions of the logic to check correctness: 2020-12-10T00:41:17.099818+00:00 <0.7> /boot/kernel.amd64/kernel: real memory = 807429734400 (770025 MB)
Diff Detail
Event TimelineComment Actions
I don't quite follow. SI_SUB_VM is before SI_SUB_CPU. Comment Actions Hmm... I got a wire crossed somewhere it seems on that. Just ignore this for the moment and I'll get back to you, will need to do some checking. Comment Actions Mark -- first, apologies for going off on the wrong track on this originally. I got the idea of the order being wrong from other triage comments and ran with that. Mea culpa. That said, digging into this more/again thanks to you setting me straight there -- I think we have a bigger, nastier bug here. vm_page_startup() builds the segments from phys_avail, sets up pages, etc. But when it calls vm_phys_init(), that path will coalesce adjacent, co-domained contiguous segments. That makes sense -- but it throws off the continue check back in vm_page_startup() that does the actual vm_phys_enqueue_contig(). I'll attach a debug boot output that I hope will illuminate this (fair warning, I've done several variations and this may have bugs in what's reported as START vs. END, please excuse any such errors in advance -- the important thing is to note the domain 1 ranges which do not get a "PHYS_ADD_SEG to DOMAIN" line - those are ones where we continued because the coalescing threw off the "entire contained in one of the ranges or doesn't overlap any of them" logic -- the segment spans the last two segments). There's a similar issue earlier in the table, but the page count is less so it doesn't stand out. I believe that this is more than a boot time print issue now (which is logical if we skip adding pages to the allocator). Looking at the vm_stats output on a different run and comparing the boot messaging: vm.stats.vm.v_page_count: 145063257 Which unless I really need more coffee and screwed up the math, is 566,653Mb -- way, way below the 730,494Mb we'd expect. Comment Actions So -- just to run this by you, because obviously I want to get something you'll like for upstream -- assuming you agree, is there a reason we don't coalesce the phys_avail array as well? I'm thinking that would be the simplest solution as it would restore the assumption in the segment walk loop. Comment Actions Just to make sure I understand how this is supposed to work:
So I think this is a real bug, but that doesn't appear to be the main issue since it should really only affect the beginning of physical memory. Rather, we are somehow getting adjacent phys_avail[] entries that can be coalesced, but the loop in getmemsize() shouldn't permit that, from my reading. Is something splitting phys_avail[] segments by calling vm_phys_avail_split() without actually carving anything out? Does OneFS have any custom vm_phys_early_alloc() calls? Comment Actions Could you please provide a dump of:
? Comment Actions No custom calls that I know of (or can find on a quick double check of the code). May end up being one I missed, of course. Comment Actions Could you please also show physmap at the beginning of vm_page_startup()? You'll need to make it a global variable, right now it's allocated on the stack. Comment Actions Since I assume the latency split may be related, I booted "-v" and pulled the SRAT lines relevant to memory (you don't need all the CPU and table stuff, I assume). Left out all the 0 addr, 0 len disabled ones as well: SRAT: Found memory domain 0 addr 0x0 len 0x80000000: enabled Comment Actions Thanks. Could you give D27613 a try? It seems the problem is that the memory affinity table derived from the SRAT may have uncoalesced adjacent entries, so when vm_phys_early_init() is called it will slice up phys_avail[] entries which should have remained contiguous. Comment Actions With D24757: SRAT: Ignoring memory at addr 0xb870000000 EARLY_SEGS: 0x0000000000200000 to 0x0000000004c39000, 19001 pages Dom 0. PHYSMAP[0]: 0x0000000000010000 to 0x000000000009f000. PHYS_AVAIL POST EARLY STARTUP: 0x0000000000010000 - 0x000000000009efff, 585728 bytes (143 pages) SEGS: 0x0000000000200000 to 0x0000000001000000, 3584 pages Dom 0. PHYSMAP[0]: 0x0000000000010000 to 0x000000000009f000. PHYS_AVAIL PRE PHYS_INIT: 0x0000000000010000 - 0x000000000009efff, 585728 bytes (143 pages) SEGS: 0x0000000000010000 to 0x000000000009f000, 143 pages Dom 0. PHYSMAP[0]: 0x0000000000010000 to 0x000000000009f000. PHYS_AVAIL AFTER PHYS_INIT: 0x0000000000010000 - 0x000000000009efff, 585728 bytes (143 pages) SEGS: 0x0000000000010000 to 0x000000000009f000, 143 pages Dom 0. PHYSMAP[0]: 0x0000000000010000 to 0x000000000009f000. VT(efifb): resolution 1024x768 Origin="GenuineIntel" Id=0x50657 Family=0x6 Model=0x55 Stepping=7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL> Structured Extended Features2=0x808<PKU> Structured Extended Features3=0xbc000400<IBPB,STIBP,L1DFL,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0xab<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 807429734400 (770025 MB) Looks right to me, but figured I'd post it here so you could look it over as well just in case. Thanks again. Comment Actions Thanks for confirming.
That's still a pretty big difference. How big is struct vm_page? Comment Actions I'd have to get the machine back to go drill down, but there's some caching allocations in there -- I don't think that's out of the norm. |