Also, please avoid filing revisions with CheriBSD tainting the diff; it can be misleading, as well as the fact that we're always a bit behind (and in this case it does matter because of the loader(8) patch landing).

arm64's parse_fdt_bootargs deals with loader(8) which will normally handle these for us when used, so we need to copy that approach given https://reviews.freebsd.org/D24912 has landed. My guess is the best place for it is parse_metadata around where we currently call init_static_kenv, to match arm64.

The first hunk is fine and should go in on its own. Again, I'd prefer a vague "prefer top-level default" / don't overdo it, folks, but whatever. It's ok.

Happy with this.

In D25531#564577, @jrtc27 wrote:

In D25531#564563, @jhb wrote:

FWIW, the general strategy on x86 and other platforms is to mask the interrupt in the PIC in the "pre_ithread" hook and later re-enable it in the PIC again in the "post_ithread" hook. Both "pre_ithread" and "post_filter" send an EOI as they are the only ones guaranteed to run synchronously on the CPU which received the interrupt. MSI alleviates the need for masking on most platforms since it is effectively edge triggered so actual level-triggered interrupts are rare on many modern systems. I'm not quite sure how that model translates to the PLIC, but on the surface reading the description of this commit seems to imply it is not doing the same thing at all. The reason we did this on x86 is that the EOI was global and could not be deferred to post_ithread. If you don't have a global EOI but instead are free to get additional interrupts (including additional lower priority interrupts) while the interrupt is not claimed, then I think your commit will work fine. However, if a high priority interrupt (PLIC priority) blocks other interrupts then you are letting that ithread starve the filters, etc. for other devices which is probably not ideal.

That's what the driver is currently doing in plic_pre_ithread and plic_post_ithread. The spec sucks and doesn't really say what claim/complete actually do, but at least the QEMU and Bluespec implementations of the PLIC will clear the IP bit and forbid the IP bit going high until completed, effectively masking the interrupt for _every_ target by masking on the _source_.

Since struct timeval is mostly obsoleted by struct timespec, I suggest adding this only when there is existing code using it.

In D25522#564549, @bcr wrote:

OK, thanks for these assessments.
@gbe: Can you either update the diff or close the PR if you think that is better than changing the existing text (just for the process= part)? For example, it could be "not a bug".

In D25312#564662, @jhb wrote:

I think the rule to only use initializers when the value is constant is probably a good compromise.

I think this is probably consistent with a fair bit of code in the tree which does use patterns like

The OCF interface looks fine to me. Supporting separate output and separate AAD is optional. For IPsec separate AAD will be more meaningful once the semi half folks land ESN support. (ESN uses a 64-bit sequence number where the upper 32-bits are implicit and never sent on the wire, so not in the mbuf.)

Update to match what I think the conensus is in this review. Please
tell me how well I did :)

Looks good from the doc side of things.

What is the next step?

Following is Makefile I was going to submit to ports some time ago:

The same port was added by me in rP540988.
Abandoning (the same as closing without committing) this.

In D25531#564579, @kp wrote:

That reminds me: the qemu implementation has a bug that means it won't trigger interrupts when we re-enable them, leading to stalls (until you trigger another interrupt, e.g. by sending something on the console).

The following Qemu patch fixes that:

It appears that Qemu is aware of this, and a patch is either already in, or on its way in: https://lists.nongnu.org/archive/html/qemu-riscv/2020-06/msg00304.html
And upon re-reading that e-mail, it looks like you may already be aware of this too...

In D25531#564590, @mhorne wrote:

In D25531#564579, @kp wrote:

In D25531#564577, @jrtc27 wrote:

That's what the driver is currently doing in plic_pre_ithread and plic_post_ithread. The spec sucks and doesn't really say what claim/complete actually do, but at least the QEMU and Bluespec implementations of the PLIC will clear the IP bit and forbid the IP bit going high until completed, effectively masking the interrupt for _every_ target by masking on the _source_.

That reminds me: the qemu implementation has a bug that means it won't trigger interrupts when we re-enable them, leading to stalls (until you trigger another interrupt, e.g. by sending something on the console).

The following Qemu patch fixes that:

--- hw/riscv/sifive_plic.c.orig	2019-11-14 19:06:20.000000000 +0100
+++ hw/riscv/sifive_plic.c	2020-07-01 15:59:38.882119000 +0200
@@ -290,6 +290,7 @@
             qemu_log("plic: write priority: irq=%d priority=%d\n",
                 irq, plic->source_priority[irq]);
         }
+        sifive_plic_update(plic);
         return;
     } else if (addr >= plic->pending_base && /* 1 bit per source */
                addr < plic->pending_base + (plic->num_sources >> 3))

Ahh, I have experienced this. It might be responsible for the deadlocks @trasz has seen running the test suite in CI. Has this patch been submitted upstream?

In D25531#564579, @kp wrote:

In D25531#564577, @jrtc27 wrote:

That's what the driver is currently doing in plic_pre_ithread and plic_post_ithread. The spec sucks and doesn't really say what claim/complete actually do, but at least the QEMU and Bluespec implementations of the PLIC will clear the IP bit and forbid the IP bit going high until completed, effectively masking the interrupt for _every_ target by masking on the _source_.

That reminds me: the qemu implementation has a bug that means it won't trigger interrupts when we re-enable them, leading to stalls (until you trigger another interrupt, e.g. by sending something on the console).

The following Qemu patch fixes that:

--- hw/riscv/sifive_plic.c.orig	2019-11-14 19:06:20.000000000 +0100
+++ hw/riscv/sifive_plic.c	2020-07-01 15:59:38.882119000 +0200
@@ -290,6 +290,7 @@
             qemu_log("plic: write priority: irq=%d priority=%d\n",
                 irq, plic->source_priority[irq]);
         }
+        sifive_plic_update(plic);
         return;
     } else if (addr >= plic->pending_base && /* 1 bit per source */
                addr < plic->pending_base + (plic->num_sources >> 3))

Any objections to this change? It's a prerequisite for using 2MB pages for per-CPU data.

Add comments requested by mhorne

In D25531#564577, @jrtc27 wrote:

That's what the driver is currently doing in plic_pre_ithread and plic_post_ithread. The spec sucks and doesn't really say what claim/complete actually do, but at least the QEMU and Bluespec implementations of the PLIC will clear the IP bit and forbid the IP bit going high until completed, effectively masking the interrupt for _every_ target by masking on the _source_.

In D25531#564563, @jhb wrote:

FWIW, the general strategy on x86 and other platforms is to mask the interrupt in the PIC in the "pre_ithread" hook and later re-enable it in the PIC again in the "post_ithread" hook. Both "pre_ithread" and "post_filter" send an EOI as they are the only ones guaranteed to run synchronously on the CPU which received the interrupt.

In D25531#564563, @jhb wrote:

FWIW, the general strategy on x86 and other platforms is to mask the interrupt in the PIC in the "pre_ithread" hook and later re-enable it in the PIC again in the "post_ithread" hook. Both "pre_ithread" and "post_filter" send an EOI as they are the only ones guaranteed to run synchronously on the CPU which received the interrupt. MSI alleviates the need for masking on most platforms since it is effectively edge triggered so actual level-triggered interrupts are rare on many modern systems. I'm not quite sure how that model translates to the PLIC, but on the surface reading the description of this commit seems to imply it is not doing the same thing at all. The reason we did this on x86 is that the EOI was global and could not be deferred to post_ithread. If you don't have a global EOI but instead are free to get additional interrupts (including additional lower priority interrupts) while the interrupt is not claimed, then I think your commit will work fine. However, if a high priority interrupt (PLIC priority) blocks other interrupts then you are letting that ithread starve the filters, etc. for other devices which is probably not ideal.

In D25530#564396, @gallatin wrote:

Why is it safe to just drop the lock?

What exactly is the LOR? I just see a stack.

Rene reported build success on CURRENT-amd64 as well. Small update, LGTM.

FWIW, the general strategy on x86 and other platforms is to mask the interrupt in the PIC in the "pre_ithread" hook and later re-enable it in the PIC again in the "post_ithread" hook. Both "pre_ithread" and "post_filter" send an EOI as they are the only ones guaranteed to run synchronously on the CPU which received the interrupt. MSI alleviates the need for masking on most platforms since it is effectively edge triggered so actual level-triggered interrupts are rare on many modern systems. I'm not quite sure how that model translates to the PLIC, but on the surface reading the description of this commit seems to imply it is not doing the same thing at all. The reason we did this on x86 is that the EOI was global and could not be deferred to post_ithread. If you don't have a global EOI but instead are free to get additional interrupts (including additional lower priority interrupts) while the interrupt is not claimed, then I think your commit will work fine. However, if a high priority interrupt (PLIC priority) blocks other interrupts then you are letting that ithread starve the filters, etc. for other devices which is probably not ideal.