Theory is to start making IPI interrupts look more like normal device interrupts. Combined with D35898 could make sense to have PPI interrupts merged with IPI interrupts instead of device interrupts.

In D38448#886009, @mjg wrote:

All this is with the assumption that it makes sense to share one intr_event struct between different cpus.

all that said, I would have to dig into what the code is doing and what makes sense to do, which I can't at the moment due to already big backlog.

Appears intr_isrc_register()/intr_isrc_deregister() would need to be after isrc_event_create(). I'm unsure whether to move the former down versus the latter up. Right now I'm mostly interested in whether this seems reasonable.

Trying for other reviewers since this has already been sitting around for months with no action.

Seems pretty reasonable to do lazy allocation of struct intr_event, but there are circumstances where it could be a disadvantage. The other likely allocation time is during the setup step, so using a flag seems appropriate. The question which comes to mind is, which flag value should be used? Should this use a high flag number since this could be discarded after the setup step? Be strictly sequential?

In D32504#912163, @hselasky wrote:

Can you rebase this patch?

And there are some compile issues (AMD64):

One concern I've got. What is the range of supported compilers? I believe some older compilers won't accept typedef struct foo type_t; as an incomplete declaration of struct foo. Those would need separate incomplete declarations, but I'm unsure whether those are included in the support window for this.

I'm really impressed at just how slow reviews for things meant to be simple are.

I am in fact taking inspiration from x86 since the general idea was to fully split the layers apart (move the outermost functions to perhaps sys/kern/subr_nexus.c). Then it might also match x86's needs and push architectures closer together. In particular, D35559 plus D39178 greatly help with merging.

Any suggestions for reviewers?

I'm distinctly unsure of the idea presently. I would like to get things better merged together, but for now other things take priority.

Now that D37938 has gone in.

Is there a committer in the house? @imp? @andrew?

At this point under "Add Action..." choose "Close Revision" to indicate it was accomplished.

Ping D38234.

I'm looking at several issues surrounding D38599 and I'm unsure which direction to go with them.

Updating to my present tree status. Problem is I see a few ways to go with this and I need advice on which to choose.

Restoring a lost hunk. Seems as some point I lost the xen_intr_pic_assign_cpu() function. Helpfully it wasn't lost from a later point.

The reasoning behind intrtab_lookup() is it should be possible to break the interrupt table functionality out of the interrupt cores. I've got some rough ideas of what such would look like, the difficulty is PowerPC being so different from everything else.

Splitting one portion off D32504 as D39178.

For the heck of it, going through and finding more instances of the pattern @zlei noticed. Seems the awk script doesn't handle the pattern ~(FLAG_TO_REMOVE) well. These are all adjust on commit, not in need of additional review.

Yet the infinite stall isn't a concern for 99.44% of systems. The two functions needing the lock exclusively are intr_event_create() and intr_event_destroy(), which most systems will only call during kernel start (and perhaps shutdown). During multiuser, those won't be called and there isn't a worry. As such sx is appropriate since an indefinite sleep isn't a concern, so this should avoid an extra unneeded restriction.

All 3 places have roughly the same level of platform specificity. subr_intr.c is INTRNG which is ARM(64)/RISC-V (nominally it could accumulate other arches, but PowerPC and x86 still have their own versions).

I agree with @zlei's comments, though this is nominally slated as a mechanical conversion so lack of cleanup could be forgiven. In other news, this does build and does appear to be usefully functional.

Since I'm not getting any other reviews, @mjg up for doing this one too?

I believe the main reason to opt for sx locks here (and the reason the x86 interrupt system uses one) is the exclusive callers are cold path. Almost solid-Helium cold. Most systems intr_event_create() will be called a number of times during kernel boot, but then never called again. As such this is a place to reduce memory use, even at the cost of substantial processor use.

In D38448#886009, @mjg wrote:

Say sizeof(u_long) == 8 and there are 8 cpus. Then this adds 64 bytes shared by all of them, where each only writes to an 8 byte portion -- meaning they all keep bouncing the same area, largely defeating the point.

Ping D37869. Lack of response really does leave the impression both D37869 and the mystery interrupt entry have been forgotten.

Updating to current tree status. Nothing particularly different, but now using intrtab_lookup().

Since D38448 presently has no reviewers, I'm nomination people who might be appropriate to review this.

Since I presently have no reviewer for D38454, @mhorne, would you be up for the job. Simply the first step of dealing with the layering issue, temporarily dump everything on architecture.

@jhb, I hope to do pretty much the same thing as D7784, but this time for the kernel portion instead of the x86 architecture portion. The reasoning is pretty well identical to D7784, this is cold-path and one portion does want to sleep, so shared-exclusive is quite appropriate. One distinction is there is no need to split any locks.

Hmm, check the source and seems I was wrong about how this worked. Update to fix issue.

Worthy of noting sys/x86/x86/intr_machdep.c also uses a shared-exclusive lock in a similar location in a similar way (hw.intrs sysctl). That seems to suggest a shared-exclusive lock is quite adequate for this use.

Updating in light of D38437. Updating based on inspiration from "hw.intrs", could be notably faster.

Update in light of D38116. The impact was interesting to figure out.

I would disagree about D38437 superseding this, but it does offer some equivalent functionality. As a result, this is indeed no longer appropriate.

D37938 has been sitting here approved, but not in the main tree for some time. This is the "D37938" branch on GitLab (git) (includes 2 associated cleanup commits).

D35527 has been sitting here approved, but not in the main tree for quite some time. This is part of the "approved" branch on GitLab (git).

Huh, Phabricator only accepted one comment being marked as done?

Candidate implementation of what @mjg requested. Adding a flag to mark interrupts which can occur on multiple processors and therefore need an array of counters.

The alternative I have is to introduce a new flag for ->ie_flags, tentatively named IE_MULTIPROC which indicates the event is triggered on multiple processors (PPI interrupt, though nominally IPI interrupts would qualify). Then turn ->ie_intrcnt into an array and allocate per-processor counters.

Was I reading the discussion on IRC correctly that padding struct intr_event to cache-line size was being suggested?

The end goal is getting INTRNG to be able to release interrupts. The goal of this series is to mostly remove intrcnt/intrnames as they're what cause trouble with releasing interrupts. For that matter the tables themselves are a layering violation and they force the architecture interrupt handling functionality to work around how they were designed.

What is here doesn't warrant the locking change. Notice though at D36610 the lock needs to be held (shared) when passing data to userspace (which can sleep). The three places the lock is needed shouldn't be used in critical sections (intr_lookup() is very slow).

Some of this network of commits could be useful without the core change. Check the "Stack" tab for the entire commit network.

I'm including this as a parent of D36610 since it results in a very handy cut-off point where the old interrupt counters can be disabled with a tiny change, before removing all of the source.

This works well with D35519/D35529/D35546. The whole stack turns to returning stray counts, which are rather more useful than 0 or EINVAL. Notably for a PPI interrupt, limited reporting of strays can be rather more consistent.

Everything split off, new state as single commit from series.

In D36610#874590, @mhorne wrote:

However, the vmstat -M core -i use-case poses a real problem for this change. This debugging feature relies on being able to read the counters from intrcnt[] and intrnames[] post-mortem. By scattering I/O interrupt counters elsewhere, it breaks this reporting functionality. Even if said feature is exceedingly niche, we can't just break userspace. It would be possible to collect interrupt counter statistics into some secondary array at panic-time, and have newer vmstat binaries read from there, but at that point I think the added complexity exceeds any tangible benefit.

I strongly dislike the approach as it is piling more complexity onto a surprisingly complex situation. While the interface definition is simple, keeping it operational is expensive. This is one of the reasons for the approach of D36610. Worse, it isn't immediately clear what the interface is, you have to look at examples in order to guess.

At this point my tree has this replaced with intrtab_lookup() since the potential for breaking the interrupt tables off the architecture interrupt cores seems valuable. Idea being to slowly pull pieces off until rather more is common.

I now expect to abandon this. At this point I've got a PoC implementation of using a resource manager to handle the interrupt number ranges. This was a thought, but it seems unlikely now.

Marking this as paused since I hope to do something about how INTRNG handles interrupt numbers. That could have a larger impact and makes this difficult. Since most interrupt controllers rarely disappear, this isn't too urgent anyway.

I think I've got a viable commit series for this, but I do hope for some feedback before creating 8 Phabricator diffs. As noted, placing the counters onto struct intr_event makes them architecture-independent, but causes them to split from the IPI counters. I think this is reasonable since this is moving the counters, not merely copying them (this is the error made at rG9b33b154b53).

Is there a time-frame for when this is desired? I was hoping to get my new build machine operational (new processor release firmware bugs, ugh) before doing another full build, but if the time-frame is short I could potentially do a test run sooner.

I don't recall what caused me to run into this. Since the original issue is no longer in the way, this no longer has sufficient value to bother with.

The name/description was hinting at the state of my tree when I noticed the issue. I suspect this will go in in a different state, update to reflect the situation better.

Guess this had shown since during build bring-up I was unsure whether these got run. No longer any value in this.

Not quite 2 years since the last word on this. The whole value to me was making it possible to build CURRENT on a RELEASE host. FreeBSD-13.1 has been released, so the value has been lost.

Then on review realize the test needs to be this instead.

Fixing copy/paste error for the panic messages.

I'm unsure how many Phabricator Diffs this will need in the end. I'm pretty sure the first few commits/patches will be:

Trying the "Foist Upon" action. I believe this should end up back in @mhorne's revision list with the review intact...

The copyright may be on the weak side for this, but this violates my rights. Notice D28958.

This is rather a bit overdue, but I note D27844 has a troublesome interaction with D16861. At fd036deac169, vmstat was modified to look for nintrcnt to distinguish kernel core dumps in which intrcnt/intrnames were arrays versus pointers. As a result right now if an ARM[64]/RISC-V kernel core dump was analyzed using vmstat the interrupt counts will be misinterpreted since it will expect an array instead of a pointer.

Hmm, reordering this means it needs this one extra bit (later on this should disappear).

I guess the goal of D36610 has moved a bit. What is really needed on this is feedback on the overall approach, then parts can be broken off as things clean up.

In D37870#869982, @fuz_fuz.su wrote:

Looks good on style alone, though I can't say much about functionality. I wonder if intrcnt, intrnames, and friends will still be needed after this change.

True enough, edit the comment too.

I've got this visible based on top of D37938 (GitLab, includes 2 cleanup commits). I would mark this approved in that state, since the functionality is reasonable to share.