Thank you very much for reviewing them! No rush to commit.

Remove previous MOVED entry.

Thanks, I'm afraid this is kind of messy however. There was a previous line on the MOVED file from the xen-{kernel/tools} -> xen-{kernel/tools}-47, and now poudriere tells me:

Fix package suffix

If it's not too much hassle please commit it :)

Apply requested fixes, run igor.

Apply requested fixes, run igor.

Thanks, all fixed :).

Thanks, new version coming.

I plan to get rid of the Xen specific MSI implementation soon.

Thanks, this allows to fix the Xen boot issue. Would you mind including the following diff into your patch?

• Bind PV IPI event channels once vcpu_id is set.

In D17013#363424, @cperciva wrote:

I wonder, can we make this work by changing the lines

PCPU_SET(vcpu_id, (regs[0] & XEN_HVM_CPUID_VCPU_ID_PRESENT) ?
       regs[1] : PCPU_GET(acpi_id));

in xen_hvm_cpu_init to

PCPU_SET(vcpu_id, (regs[0] & XEN_HVM_CPUID_VCPU_ID_PRESENT) ?
       regs[1] : -1);

and then resurrecting xen_set_vcpu_id but having it only set vcpu_id if it is currently set to -1?

That way if we have XEN_HVM_CPUID_VCPU_ID_PRESENT we'll get vcpu_id from there (and it doesn't matter if that runs before acpi_id is set) and if we don't then we'll fill in the right value slightly later (after acpi_id is set, but we don't need to be running on the AP in question at that point).

In D17001#362843, @jhb wrote:

In D17001#362597, @royger wrote:

In D17001#362583, @kib wrote:

Would be useful to assert that there is no IO APICs or AT PICs configured when num_io_irqs == 0, but I am not sure that this is straightforward.

I thought about this too, but I didn't find any straightforward way to check whether there are any PICs in the system. I will wait for John in case he has a suggestion.

There isn't a good way to assert that. The ATPIC code always assumes it can use the first 16 IRQs if 'device atpic' is enabled in the kernel which is a bit of a landmine. 'device atpic' isn't in GENERIC, but it might be nice to force it to be disabled for this particular Xen case? Or is there an emulated ATPIC with no useful interrupts? (In a kernel without 'device atpic' we use unconditional logic to init the 8259A's into a silent state.) If Xen is still providing a dummy set of 8259A's, then I think what I'd actually prefer is to patch the SYSINIT in local_apic.c that does 'apic_setup_io' to bump num_io_irqs up to 16 if it is still zero to avoid any possible conflicts with IRQs 0 - 15. Note that one idea I'm considering is removing MINIMUM_MSI_INT post-branch so that MSI IRQs would start at the end of the PIC range rather than at 256 in which case explicitly reserving 0-15 would still matter.

• Remove xenpv_set_ids.
• Make sure acpi_id is set when starting APs.

In D17001#362583, @kib wrote:

Would be useful to assert that there is no IO APICs or AT PICs configured when num_io_irqs == 0, but I am not sure that this is straightforward.

In D17000#362586, @kib wrote:

In D17000#362582, @royger wrote:

They are not too large, what I see is: "contigmalloc - size: 0x1000 low: 0x1000 high: 0xa0000", which should be fulfillable. I guess there's something else that allocates this memory, but I have no idea of what yet. This is probably a glitch of running as a PVH guest, which has no BIOS and no reserved regions < 1MB.

Ah so it is low memory. Does it make sense to init bios at all, on such machine ?

• Add newline at the start of x86bios_unmap_mem.
• Use consistent order for the flags.

In D17000#362579, @kib wrote:

This is in fact worrying. How large are these requests ? If they cannot be satisfied at the early boot stage, then they cannot be at all, I think.

I'm going on vacations for the whole next week and I'm not taking any computer, so I won't be able to commit this. If someone feels like picking this up in the meantime that's great, if not I will pick it up in two weeks.

Remove unrelated netfront change.

Tested with Xen on amd64, no issues. Thanks.

The following diff on top of your patch seem to fix the issue:

diff --git a/sys/x86/xen/xen_intr.c b/sys/x86/xen/xen_intr.c
index 4b64ac36b774..c7328ad71e27 100644
--- a/sys/x86/xen/xen_intr.c
+++ b/sys/x86/xen/xen_intr.c
@@ -634,17 +634,13 @@ xen_intr_init(void *dummy __unused)
 	mtx_init(&xen_intr_isrc_lock, "xen-irq-lock", NULL, MTX_DEF);

This panics quite early with the following trace:

/boot/kernel/kernel text=0x167caa8 data=0x1cdac8+0x79c1f0 syms=[0x8+0x1803c0+0x8+0x19cd3e]
/boot/entropy size=0x1000
Booting...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-ALPHA2 #0 d016733ea(master)-dirty: Thu Aug 23 13:55:07 UTC 2018
    root@freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
XEN: Hypervisor version 4.12 detected.
CPU: Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz (3504.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
  Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c6fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x1000
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 16768827392 (15992 MB)
avail memory = 16202080256 (15451 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Xen HVM>
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: L2 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: L3 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 cache groups x 1 core(s)
random: unblocking device.
panic: Assertion intrcnt_index < nintrcnt failed at /usr/src/sys/x86/x86/intr_machdep.c:468
cpuid = 0
time = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff82696900
vpanic() at vpanic+0x1a3/frame 0xffffffff82696960
panic() at panic+0x43/frame 0xffffffff826969c0
intrcnt_add() at intrcnt_add+0xc1/frame 0xffffffff826969e0
xen_intr_init() at xen_intr_init+0xd0/frame 0xffffffff82696a50
mi_startup() at mi_startup+0x118/frame 0xffffffff82696a70
btext() at btext+0x2c
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
db>

Ping?

So it seems there are still issues even after this fix, now I'm seeing:

Thanks, that look great!

I plan to commit this tomorrow unless there are any objections.

Thanks! LGTM, just one minor issue with the command line because we need to use dom0pvh=1 when using the xen47 packages and dom0=pvh when using the xen411 packages.

Change the wording of the comment and put it just before the first
usage of DB_FROM_SRC.

In D16136#350352, @pratyush wrote:

Probably a difference in the tab width. I don't get why there is indentation with both tabs and spaces in a single line. The entire blkfront code is like that. Some places its all tabs, some places it is a mix of tabs and spaces for indentation. Its weird and confusing.

The code is OK, but the coding style is not. I had to fix the lines to be < 80 cols and adjust the indentation.

LGTM.