Avoid sleeping after a failed vm_page_alloc_contig() call during boot.

[markj]

Summary

After r297466, a VM_WAIT during boot will result in a hang since the
pagedaemon may not yet be running. Previously, VM_WAIT would just
return immediately.

I tried to address one instance of this issue with r302147, but there
are others in vesa(4), for instance. It seems to me that we should just
avoid the VM_WAIT if the kernel is still booting to avoid these types
of regressions after r297466.

[bdrewery]

We've been hitting boot hangs in vesa load due to the VM_WAIT sleeping on small VMs. This patch resolves the issue.

[alc]

What are the characteristics of the physical allocation by vesa?

[markj]

In D7417#154228, @alc wrote:

What are the characteristics of the physical allocation by vesa?

It uses x86bios_alloc(), which always attempts to allocate from the first megabyte of physical memory. In our case, I believe it's trying to allocate a single page.

[alc]

In D7417#154234, @markj wrote:

In D7417#154228, @alc wrote:

What are the characteristics of the physical allocation by vesa?

It uses x86bios_alloc(), which always attempts to allocate from the first megabyte of physical memory. In our case, I believe it's trying to allocate a single page.

How much pseudo-physical memory does the virtual machine have? During initialization, we shouldn't be allocating from the first 16 MB of physical memory except when it's explicitly asked for or the total physical memory is so small that there is no other option. I suppose that I'd like to see the output from "sysctl vm.phys_free vm.phys_segs" on the affected machine, because I'm confused. :-)

[markj]

In D7417#154261, @alc wrote:

How much pseudo-physical memory does the virtual machine have? During initialization, we shouldn't be allocating from the first 16 MB of physical memory except when it's explicitly asked for or the total physical memory is so small that there is no other option. I suppose that I'd like to see the output from "sysctl vm.phys_free vm.phys_segs" on the affected machine, because I'm confused. :-)

These have 2GB of memory. I should note that this doesn't happen consistently: our build automatically creates and boots a VM, and it occasionally boots successfully without this change. So bdrewery and I should probably determine why that's the case first.

vm.phys_segs:
SEGMENT 0:

start: 0x1000
end: 0x9c000
domain: 0
free list: 0xffffffff80e9d730

SEGMENT 1:

start: 0x100000
end: 0x200000
domain: 0
free list: 0xffffffff80e9d730

SEGMENT 2:

start: 0x407e000
end: 0x40c0000
domain: 0
free list: 0xffffffff80e9d730

SEGMENT 3:

start: 0x40c1000
end: 0x7be91000
domain: 0
free list: 0xffffffff80e9d730

SEGMENT 4:

start: 0x7ff00000
end: 0x7ffe0000
domain: 0
free list: 0xffffffff80e9d730

vm.phys_free:
DOMAIN 0:

FREE LIST 0:

ORDER (SIZE)  |  NUMBER
              |  POOL 0  |  POOL 1

• -- -- -- -- -- 12 ( 16384K) | 0 | 0 11 ( 8192K) | 0 | 0 10 ( 4096K) | 0 | 0 9 ( 2048K) | 0 | 0 8 ( 1024K) | 1 | 0 7 ( 512K) | 3 | 0 6 ( 256K) | 10 | 0 5 ( 128K) | 15 | 0 4 ( 64K) | 43 | 26 3 ( 32K) | 43 | 62 2 ( 16K) | 79 | 91 1 ( 8K) | 298 | 172 0 ( 4K) | 1841 | 572

[alc]

I don't see a "FREE LIST 1:". That's worrisome. The physical memory below 16 MB should be in a different set of free lists.

Once upon time, Isilon was having problems with, I believe, USB devices and allocations below 4 GB, and I gave Jeff a hack to repurpose the mechanism used to protect memory below 16 MB. I hypothesize that you're still using this hack, and that it is responsible for your problems. Stock FreeBSD has a different and better fix that deals with both the 4 GB and 16 MB cases simultaneously. Please check your vm_phys_init() against the stock version.

[alc]

The other thing to check is amd64/include/vmparam.h. Do you see:

/*
 * Create up to three free page lists: VM_FREELIST_DMA32 is for physical pages
 * that have physical addresses below 4G but are not accessible by ISA DMA,
 * and VM_FREELIST_ISADMA is for physical pages that are accessible by ISA
 * DMA.
 */
#define VM_NFREELIST            3
#define VM_FREELIST_DEFAULT     0
#define VM_FREELIST_DMA32       1
#define VM_FREELIST_ISADMA      2

[markj]

In D7417#154272, @alc wrote:

I don't see a "FREE LIST 1:". That's worrisome. The physical memory below 16 MB should be in a different set of free lists.

I think I see the problem. We reverted our changes to vm_phys_init() after (I believe) r276439, but we still have:

120 #ifdef VM_FREELIST_ISADMA
121 /* Begin Isilon */
122 #define VM_ISADMA_BOUNDARY 4LL * 1024 * 1024 * 1024
123 /* End Isilon */
124 #endif

in vm_phys.c. That probably should have been dropped as well.

[markj]

Fixing that seems to have done the trick. Thanks Alan.

markj-1# sysctl vm.phys_free
vm.phys_free: 
DOMAIN 0:

FREE LIST 0:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1
--            -- --      -- --      --
  12 ( 16384K)  |       0  |       0
  11 (  8192K)  |       0  |       0
  10 (  4096K)  |       0  |       0
   9 (  2048K)  |       0  |       0
   8 (  1024K)  |       2  |       0
   7 (   512K)  |       1  |       0
   6 (   256K)  |      22  |       0
   5 (   128K)  |      44  |       8
   4 (    64K)  |     116  |      14
   3 (    32K)  |     190  |      17
   2 (    16K)  |     466  |      30
   1 (     8K)  |       0  |      15
   0 (     4K)  |       1  |      33

FREE LIST 1:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1
--            -- --      -- --      --
  12 ( 16384K)  |       0  |       0
  11 (  8192K)  |       0  |       0
  10 (  4096K)  |       0  |       0
   9 (  2048K)  |       0  |       0
   8 (  1024K)  |       0  |       0
   7 (   512K)  |       0  |       0
   6 (   256K)  |       0  |       0
   5 (   128K)  |       0  |       0
   4 (    64K)  |       0  |       0
   3 (    32K)  |       0  |       0
   2 (    16K)  |       0  |       0
   1 (     8K)  |       0  |       0
   0 (     4K)  |       3  |       0