This is acceptable to me, but I believe the desire is have an okay by @markj. I believe resigning should make this more visible to @markj, rather than accepting.

I think the usual is to let the differential owner do the update. Otherwise the steps end up being "Commandeer Revision", update, "Foist Upon".

Most of what is presently here is definitely wrong. xen_intr_handle_upcall() retrieving the trapframe from curthread->td_intr_frame is definitely right. Issue is xen_intr_handle_upcall() needs to behave as a proper driver_filter_t for the XenPCI support. ARM(64) and RISC-V also need this, likely PowerPC would need similar. I do see two ways forward though:

I think so (suddenly worried about famous last words).

This is oddly similar to what Julien Grall did in 2015. For aarch64 the suggested range for the shared information page is available to the platform device. Perhaps the architecture should pass the struct resource * to xenpv?

If you check your archive of e-mail you will find I expressed uncertainty over whether curthread->td_intr_nesting_level should be incremented/decremented. Removing that is fine, but the rest is wrong.

So, most approaches to getting interrupt subsystems to converge are going to need something along the lines of D35559 at some point. Might it be possible to get a decision/activity on D35559?

I had been thinking @jrtc27 was supposed to remove her "changes requested" tag, yet I was advised she thought @mmel should instead review the replacement. So, still need a decision as to whether the approach is D43980 -> D40474, versus intr_pic_init_secondary() only calling for FLAG_PIC entries.

Update to better reflect current state of FreeBSD tree. My repository had several patches, one of which had modified intr_describe(). As such this better reflects current state.

Splitting D40166 back up. While I like the second part, it is lower value. This would serve to make all architectures noticably more similar.

Change the approach and use ->is_event->ie_irq for places besides intr_register_source(). Since D35386 is now a parent, the vector value can disappear from xen_arch_isrc_t. msi_irq can also disappear, but that is more involved.

Due to D36901 getting in first, intr_bind() disappeared. As threatened, I really like intr_add_handler()'s first two arguments being in the opposite order. Not only does this seem more consistent with the rest of the interface, but this also matches INTRNG (the domain argument is now the only difference).

This is another approach to the issue pointed out in D40474, mainly using ->pic_flags as flags and allowing a pic_list entry to fill both roles.
Another variant of this strategy would be to add a pair of boolean arguments to intr_pic_register() so the caller would indicate which roles they can fill. This variant could be argued to be better as each layer of the implementation could get its own pic_list entry.

Depending on where xen_intr_disable_source() and xen_intr_disable_intr() were being called from, this is fixing both 1797ff962769 and some other commit. Certainly 1797ff962769 may have exposed this, but the is_valid_evtchn() call was needed earlier (could be rG76acc41fb7c7, or even before).

In D40474#1001982, @jrtc27 wrote:

I'm really struggling to understand what the source of confusion here is.

@jrtc27 not being on intimate terms with the GICv3 implementation and lacking hardware to confirm how things work, I've got no idea what to do. What I do know is the extra PIC for this device never gets into sc->gic_children and therefore the loop in gic_v3_init_secondary() doesn't work for this device.

@jrtc27 I was hoping for an answer to my question before doing anything.

Uploading the obvious SLIST->STAILQ adjustment.

Guess I had it easy as I had a system running Linux and arcanist was available as a package.

In D40474#993115, @jrtc27 wrote:

You want if (dev != intr_irq_root_dev) return; added to every preexisting pic_init_secondary() implementation? Luckily this is simple as there are very few implementation (bcm2836 is the only one which wasn't mentioned).

That's neither necessary nor sufficient, so no.

In D40474#993106, @jrtc27 wrote:

Whilst this looks like the right thing to do for the core framework, I believe our GICv2 driver won't like this in some edge cases, as for the FDT case it supports being a non-root PIC yet has an arm_gic_init_secondary which currently only runs for the root PIC, probably by design?

If in the "Accepted" state (someone selects "Accept Revision") the option "Close Revision" appears underneath "Revision Actions". I'm unsure whether there are other ways to get that to appear. Isn't available here since no reviewer ever marked it as such.

The only remaining usage of this is sys/arm/arm/pmu.c. If that can get removed then it become possible to use NULL as a handler argument. I can turn that removal into a review/differential if someone desires, but seems trivial enough to simply commit if someone has a chance.

In D42343#969231, @royger wrote:

My opinion is that halt should behave equally on bare metal than when virtualized, otherwise it's just going to confuse users. If we believe that halt should behave differently (iow: do power off) it should be changed uniformly.

I think I've found a better way to address this. Alas tis a bunch of commits and the throughput for non-sponsored patches is atrocious.

I mostly concur with D42343, I've noticed the behavior and found it suboptimal. Only trick is after a fair bit of time not looking at this I found a small issue which needs to be addressed.

Marking D32632 as abandoned. I've got something which approaches the same goal, but is distinct enough to not take over D32632.

Hmm, lost the mention of isrc_event_destroy(). I suppose if one wants to have that function, things could look like this instead.

Due to D36901 getting in first, intr_bind() has disappeared and the portion of D35386 adjusting the interface disappears.

When it comes down to it, since I'm more concerned with getting rid of sys/kern/kern_intr.c:intr_lookup() it is acceptable to continue having intr_{get|set}affinity() using numeric interrupt numbers. This will require sys/kern/kern_intr.c to #include the architecture interrupt headers. I was reluctant to propose that due to how distinct they are, but this doesn't actually contradict my goals and is nominally acceptable to me.

Tiny adjustment, split the NULL-check to a separate step. Just in case a use does the NULL-check itself and simply wants the appropriate field name. Hopefully common-subexpression elimination would take care of this, but it might not.

All of these should be obvious from what INTRNG does. Failing that, an example was provided. So, a few call sequences:

I'm unsure whether the call to child devices during gic_v3_init_secondary() should be removed. The only user of this is the GIVv3 ITS (550d01a2117f). Will most child devices need this called after the parent PIC is fully initialized? Do child devices merely need this called during processor bring-up?

Looks like sys/arm/mv/gpio.c has a situation similar to what I'm looking at. The approach of doing local calls to intr_event_create() has been tested and functions, but this seemed unlikely to be accepted by a reviewer. Yet this makes me the second person to run into troubles with INTRNG due to assumptions in its interface.

This wasn't my goal, but it looks like this series, D38451 in particular fixes the intr_isrc_dispatch() call in sys/arm/mv/gpio.c. Since the counters get moved to the event, it ends up with functioning counters, plus no uninitialized counters get incremented (the function mv_gpio_intr_handler()).

This appears to partially fix the intr_isrc_dispatch() call in sys/arm/mv/gpio.c, since it removes the increment of the counters which have never been initialized.

Alas this came to my attention way to late to address on-time. Callers of intr_event_destroy() want to know whether everything has been appropriately released or not. If the event is NULL this likely something failed very early during setup. In such case the event doesn't need to be released since it was never allocated. As such 0 should be the return instead.

In D33818#771982, @kib wrote:

So provide the patch that fixes that 'many' places.

Squash review, adding removal of ->isrc_irq. This is one reason why allocating the event earlier has value, ->ie_irq can then substitute everywhere. If reviewers want this split back off, that can be done.

Hmm, no that isn't really so elegant so keep to the smaller diff.

Hmm, spotted this a while ago, but didn't get Phabricator updated. The test in isrc_event_create() was wrong, this appears to be the correct action.

Having examined things more, an appropriate place to store the value already exists, but has been underutilized. Mainly isrc->is_event->ie_irq.

D40166 heads in the opposite direction with regards to the interrupt number. Instead of removing it from intr_event and pushing it back to the architectures, D40166 allows removing it from the architecture and only having it on intr_event.

Move the setting line to isrc_alloc_irq(). This would allow keeping the interrupt number strictly on the event and removing it from intr_irqsrc.

One extra may be needed for this. The file sys/arm/mv/gpio.c has a call to intr_event_create(). Unfortunately this isn't setup for INTRNG allocating events early. Interestingly it seems ad2be10ff043 is exactly the same issue I'm running into.

In D32504#912405, @gallatin wrote:

I'm not sure what the point of this is. To simplify use cases like hotplug by giving intr API consumers a way to call intr_foo() functions without keeping track of the vector?

I've been concerned over whether the initial version of D40166 would pass. The use case was too specialized. As such an update, remove the lazy interrupt allocation.

In D38599#944958, @markj wrote:

Absent some examples grounded in a specific driver to motivate the discussion, my view is that it's better not to bother thinking about this at all. If intr_event_destroy() returns EBUSY in a particular scenario, then the caller should be fixed.

A handy example is D38599. It could test isrc->isrc_event before calling intr_event_destroy(), but if intr_event_destroy() will gracefully handle ie == NULL then it is easier not to bother. If you prefer to have a panic() sooner rather than later, the

if (ie == NULL)
        return (EINVAL);

should be taken out and hopefully a panic() will ensue on line 539. If intr_event_destroy() is going to check for ie == NULL then it should do something useful if that occurs. Returning EINVAL is almost certain not to be useful and thus returning 0 seems more sensible.

Errf, then find the goof after push.

Update to get things right. Though this is almost a fix on commit situation.

Update as per comments.

In D41462#944302, @emaste wrote:

I believe CPU IDs are not guaranteed to be contiguous and mp_maxid is what should be used instead

In light of D41319 and the issues of what was observed when similar was done there. Presently this is untested, but I'm reasonably sure this is correct.

I hadn't realized there was an actual sys/arm/arm/generic_timer.c which appears to be used for ARM64. As such I've identified 3 places on ARM where a single intr_event structure is used for interrupts which occur on multiple processors. I now fear this could be all over ARM and this may already have been endorsed by fiat.

I'm unsure whether it works well as part of any existing differential, but it was suggested there should be a cpu_t type definition. That didn't fit as part of D36901, but does seem worthwhile. That would likely address the breakage which was found.

"My kingdom for a review"?

This looks plausible. Likely it mostly amounts to a debugging tool since you would basically never want this in anything approximating a real world situation.

When it comes down to it, 9b33b154b53 was pretty much a simple hack to allow architectures without a proper interrupt table implementation to do simple interrupt number lookup. Thing is, MIPS was the last architecture lacking such a table, therefore this is now fully redundant with architecture functionality. As such this is effectively cleaning out yet another MIPS remainder.

Unfortunately see D38448 for most comments so far.

Updating D38448 to split off the counter addition. At this point I'm under the suspicion the core team needs to consider this, and say "yay" or "nay". Whether interrupts which occur on multiple processors should share an intr_event structure is a real question. So far several places appear to be saying "yes", but I would like a core team review.

This is splitting the counter add commit off the flag add commit. Originally these seemed like a single task, but seems further consideration is needed for the flags.

Uhm, this would be kind of useful to have. Any chance this can get a review?

Missed it when looking for a fix in recent history. Most recent build succeeded, so looks like this was the fix.

Latest build attempt failed. This is my current suspect.

Rebuild with 8920c5f2a11 reverted succeeded. Looks like D41320. The host is FreeBSD 13.2p2 with minimal extras.

I'm starting to think the flag addition to he lower layers should be broken into a separate review. Issue is that has some effects all over the place.

At the end of a full build which just terminated:

ld: error: unable to find library -lsoxstack
clang: error: linker command failed with exit code 1 (use -v to see invocation)
--- stack_dt_need_exec_test.full ---
*** [stack_dt_need_exec_test.full] Error code 1

Looks like this broke the build.

Updating to re-add a commit which got dropped.

Updating, spreading the proposed flag further. Still need an answer for whether multi-processor interrupts should share an intr_event structure though.

Appears INTRNG isn't the only thing tending to push multi-processor interrupts onto single intr_event structures. At D25754 a multi-processor soft interrupt was created. As such it would need to share this.

Alerting everyone this is on the horizon. A check 3 weeks ago showed there are 3 remaining places where this is utilized. I've got patches out trying to get them nuked while bypassing Phabricator (they're trivial), but am presently waiting on news. Those are in the files sys/arm/arm/pmu.c and sys/x86/xen/xen_apic.c.

In D35976#927426, @emaste wrote:

We seem to use ldscript as either an extension or base for the filename, e.g. sys/conf/ldscript.arm64 or lib/libc/libc.ldscript.

Huh, I was so focused on interactions with @royger I missed that I've done rather substantial changes to what was D30996.

Unfortunately there are several overlapping issues interwoven here and choices need to be made.

Note on this. Should this perhaps be a separate file in sys/kern instead of mixed in with INTRNG/subr_intr.c?

In D38599#926723, @markj wrote:

In D38599#923991, @ehem_freebsd_m5p.com wrote:

With no activity, hopefully doing an update will provoke activity.

Most of the kernel appears to treat NULL as unallocated and not an error.

Are you referring to functions like free()? It depends. free() is called very frequently in error handling paths, so having free(NULL) be a no-op is a convenience. Most rarely used allocators do not have such an affordance. uma_zdestroy(NULL) is a bug, as is kmem_free(NULL). I think this is the right tradeoff, since the permissive behaviour can hide bugs, and calls to free() or m_free() are a lot more common than uma_zdestroy() or kmem_free().

With no activity, hopefully doing an update will provoke activity.

Pulling the intr_irq_root_dev == NULL check. There are similar checks elsewhere and that won't directly effect this spot.

In case anyone is wondering, to answer the comment "QQQ: Only root PIC is aware of other CPUs ???". That is indeed the way this was implemented. The more important issue is are PICs besides the root PIC interested in being informed when processors are brought on-line.

Having considered things in various places I'm no longer so sure of this being a good idea. There is merit, but due to a certain case I could see this being reused for major gain. As such marking this as on hold.

Check the build and naturally find oversights from the quick implementation. Tiny fix. One item is whether the pic_list_lock should in fact be held...