I am sorry to speak up only now, after the review sat open for so long. But in looking over the change it seems to be incomplete.

I added MPASS macro following the example of AMD64's code in a corresponding place. I verified that the execution always visits ecall_handler() before it arrives here.

if td1 is always curthread, then a kassert is needed

how hard it would be to implement MSI interface here? to support MSI interrupts from devices, which is a crucial functionality

In D53232#1216532, @jrtc27 wrote:

This will not fix that bug. fence is fence iorw, iorw, so all you are doing is relaxing some ordering requirements.

This will not fix that bug. fence is fence iorw, iorw, so all you are doing is relaxing some ordering requirements.

@jrtc27 I've update the revision, PTAL

In D53232#1216063, @jrtc27 wrote:

Do we actually have a proper definition of what *our* memory barrier APIs are meant to mean? What the ISA manual says is a sensible mapping isn't necessarily what we want; note that the table you're referencing is for "Linux memory primitives", and we are not Linux, nor do we follow its memory model.

Do we actually have a proper definition of what *our* memory barrier APIs are meant to mean? What the ISA manual says is a sensible mapping isn't necessarily what we want; note that the table you're referencing is for "Linux memory primitives", and we are not Linux, nor do we follow its memory model.

Shall I send a new patch? No comments on this one.

FYI, it's best practice to generate diffs with full context (-U999999 or whatever).

• Refactor aplic code into common init and aplic/imsic specific code.
• Add support for IMSIC.

In D43452#1055467, @br wrote:

Hi Himanshu what is status of this? could you regenerate for the latest HEAD ?

Hi Himanshu what is status of this? could you regenerate for the latest HEAD ?

Reworked APLIC_IDC_REG as asked

Sorry for the mess up in versions in comments. Last patch was v9, as per my local branches. This one is v10.

Changes in v10:

• Save hart indices from FDT and use them to create target value

LGTM!

ping.

After Convert local interrupt controller to a newbus PIC, commit rebase is required.

Changes in v7:

• Keep per-cpu IDC offsets after reading interrupts-extended property
• Accessing IDC is via the stored per-CPU IDC offsets
• Remove cleanup after failure in *_attach function (like other drivers)
• Added aplic.c in files.riscv alphabetically

In D43293#993161, @himanshu_thechauhan.dev wrote:

In D43293#993109, @jrtc27 wrote:

In D43293#991986, @himanshu_thechauhan.dev wrote:

@jrtc27 Thanks for your review! I have taken care of your comments. Patch will follow shortly.

Unless I'm mistaken you still use the hartid to index the APLIC, not the APLIC's hart indexes. You need to actually record, and later use, the index <-> cpu mapping when iterating over interrupts-extended.

Whatever hartid is read from the FDT, its logical CPU is kept in target_cpu cpuset. We already, in a way, keep that mapping. When the interrupts are bound, the logical CPU from target_cpu points to the required hart. There is no need to keep separate mapping.

In D43293#993109, @jrtc27 wrote:

In D43293#991986, @himanshu_thechauhan.dev wrote:

@jrtc27 Thanks for your review! I have taken care of your comments. Patch will follow shortly.

Unless I'm mistaken you still use the hartid to index the APLIC, not the APLIC's hart indexes. You need to actually record, and later use, the index <-> cpu mapping when iterating over interrupts-extended.

In D43293#991986, @himanshu_thechauhan.dev wrote:

@jrtc27 Thanks for your review! I have taken care of your comments. Patch will follow shortly.

Changes in v6:

• Added verification of harts that can take interrupt using "interrupts-extended" property of FDT.
• Added target_cpu cpuset which contains the bit map of logical cpus that can take interrupts.
• During binding of the irqs, the cpus from target_cpu cpuset is used.
• Handled failure case and deregister ircs.
• Took care of comments regarding the brackets.
• Added a macro APLIC_IDC to give offset of a given hart.
• Removed return of FILTER_STRAY when IRQ is 0, added a KASSERT instead.
• Use of DEFINE_CLASS_0

@jrtc27 Thanks for your review! I have taken care of your comments. Patch will follow shortly.

In D43293#991694, @jrtc27 wrote:

Also this doesn't properly deal with the interrupts-extended property. I believe that, like the PLIC, you need to iterate over its cells to figure out the mapping between architectural hart ID and APLIC hart index, since they are two independent schemes.

With apologies for the delay:

@mhorne That's great! Thanks for all your time and guidance! I really appreciate it!

I am happy with it, and did some testing locally. Rebinding interrupts seems to work as expected.

changes in v6:

• Fixed wrong sizing of isrcs
• Fixed indentation of goto label

This looks pretty good, with one remaining issue that I see.

Changes in v5:

• Use hartid instead of logical cpu when programming APLIC registers
• Add defines for some hard coded numbers
• Check for "msi-parent" property during attach and fail if found
• Removed the IDC structure and added defines for the IDC register offsets and calculations
• Removed reuse of local variables
• Removed unused function and structures
• Removed "all rights reserved" from copyright comment
• Removed ambiguity of cpu vs hartid

@mhorne Thanks for the review! I have taken care of your comments. Will be posting new patch shortly.

Thanks a lot for submitting this!

FYI I intend to give this a more proper review this week, now that I've returned from my travels over the holidays

v4

• Fixed bug
  • Uniprocessor boot used to hang.
  • Now it is tested from 1 CPU to 16 CPUs
  • virtio block dev is tested
  • virtionet is tested

v3 changes:

• Removed duplicated defines
• Added comments to some of the structures and defines
• Cosmetic changes.

tools/build/checkstyle9.pl will check most common style issues

Took care of formatting and other comments.

In D43293#986788, @jrtc27 wrote:

I have a related series ending in https://reviews.freebsd.org/D35901 to get back to that should make having direct and IMSIC APLIC support less messy due to changing what the root PIC is. That shouldn’t block this patch as it’s (presumably) just similar tweaks to the normal PLIC, but for IMSIC support it’s probably worth me pushing forward on that before you get too far with it?

I have a related series ending in https://reviews.freebsd.org/D35901 to get back to that should make having direct and IMSIC APLIC support less messy due to changing what the root PIC is. That shouldn’t block this patch as it’s (presumably) just similar tweaks to the normal PLIC, but for IMSIC support it’s probably worth me pushing forward on that before you get too far with it?

Time to teach my emacs the style(9) KNF. I will take care of these suggestions. Thanks!

Note that some lines don't fit 80 chars per style (9) and some have too much whitespace after wrap.

Interesting!

In D39884#930242, @jrtc27 wrote:

In D39884#930234, @christos wrote:

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

Can you recompile kinst with this patch applied? https://reviews.freebsd.org/P592
If we run it with a single trampoline per page, the crashes seem to go away.

Is it just a case of needing to smp_rendezvous a fence_i since it's not broadcast by the hardware like on arm64?

In D39884#930242, @jrtc27 wrote:

Is it just a case of needing to smp_rendezvous a fence_i since it's not broadcast by the hardware like on arm64?

In D39884#930234, @christos wrote:

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

Can you recompile kinst with this patch applied? https://reviews.freebsd.org/P592
If we run it with a single trampoline per page, the crashes seem to go away.

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()