Differential D23376
Avoid having PowerMacs ending up with stuck-sleeping threads: force some boot-time TB value relationships across sockets/cores. Authored by markmi_dsl-only.net on Jan 26 2020, 11:04 PM. Tags Referenced Files
Details
Explicitly initialize the bsp's TB value early on. For PowerMacs with multiple sockets or cores only, have machdep_ap_bootstrap for each ap make a round trip with the bsp to get a current bsp's TB value and then shift the ap's time so that the midpoint of the shifted round tip interval approximately matches the TB value the bsp supplied. The round trips and data transfers use a memory protocol. For non-PowerMacs, do the old way for the ap's. I've been running based on the patches being invovled since late 2019-May or so: PowerMac11,2 with 2 sockets, 2 cores each, and PowerMac3,6 with 2 sockets, 1 core each. No examples of 1 socket with 2 cores, unfortunately. No examples of powerpc64 or powerpc non-PowerMacs either. I do use the same source tree to build for amd64, aarch64, and armv7 and they have been working too. But the patch material is extracted from a source tree that has more local changes: the testing has not been with only these changes. I've not had any examples of the stuck-sleeping issue since adopting these changes. Nor have I had new problems from the changes.
Diff Detail
Event TimelineComment Actions Note: I was not (and am not) style(9) literate when I did the investigation that eventually stabilized on this code. My information-handling capacity was limited. I fully expect this will need to be dealt with at some point. But I'd prefer to first have the technique(s) used be thought to be stable before focusing on the other type of knowledge. I'm also okay if my material inspires something and the fix does not end up being my code. FYI: The submittal was based on a head -r356426 context. Comment Actions If there are non-PowerMac powerpc platforms that get the threads-stuck-sleeping Also, I was only targeting being sufficiently accurate across sockets/cores to avoid the My own evidence of PowerMac sufficiency for my intended purpose of avoiding the PowerMac11,2 2.5 GHz PowerPC 970MP 2 sockets, 2 cores each I've no evidence yet of insufficiency for these. All of them previously had If this does not work as well for the fuller range PowerMacs as I expect, Comment Actions Yeah, I used to see this too (on either power9 or amigaone x5000, can't remember which.) IIRC, the problem is that if the tb is significantly unsynced, the callwheel algorithm falls over and misses wakeups for things. Comment Actions I'd only seen PowerMac reports. "used to": has it been fixed for where you saw it? If it still can be a problem outside PowerMacs, then an "alternate_timebase_sync_style= 1;" Yea, the end result is that sleepq_timeout ends up doing: if (td->td_sleeptimo > sbinuptime() || td->td_sleeptimo == 0) {
/*
* The thread does not want a timeout (yet).
*/
} else . . .and that leads to never trying again in the overall call structure: no It makes sense to me that FreeBSD does require some scale of matching
Comment Actions What we really want to achieve is a better timebase sync mechanism. This can only be achieved by disabling the timebase, rendezvousing, and re-enabling it. Adding a drift into the generic code for a specific platform is not the correct solution. I do intend to shortly (next few days) clean up the one additional platform_timebase_sync() user (cpu_sleep()) to not use it, and handle it directly, since the code in cpu_sleep() is actually mpc74xx-specific, it's not generic at all. So once that's made explicit, a proper rendezvous sync should be done in platform_powermac.c instead. You can get very close by mimicking what I do in platform_mpc85xx.c, by using atomics to handle the rendezvous, and doing a multi-stage rendezvous.
Comment Actions Speaking in terms of the mpc85xx code: At this point, I've no clue how to find the information for setting up various(?) alternatives to the mpc85xx_rcpm_freeze_timebase/mpc85xx_guts_freeze_timebase implementations as far as what each would need to do or how many alternatives would be needed to cover everything. Also, I've only got access to some old dual-socket PowerMac G4s and G5s to confirm any tentative understanding with.
Comment Actions You can look in Linux's arch/powerpc/platform/powermac/smp.c, and arch/powerpc/kernel/smp-tbsync.c for how Linux does it. Their generic tbsync handshake is similar to yours.
Comment Actions Looking shows 4 uses of platform_smp_timebase_sync in the -r358132 powerpc code from svn: static void cpu_mp_unleash(void *dummy) does: platform_smp_timebase_sync(ap_timebase, 0)
void machdep_ap_bootstrap(void) does: platform_smp_timebase_sync(ap_timebase, 1)
void cpu_sleep() does: platform_smp_timebase_sync(timebase, 0)
void cpudep_ap_setup() does: platform_smp_timebase_sync(0, 1)That last has a comment reporting: /* The following is needed for restoring from sleep. */ Are you also removing the usage from cpudep_ap_setup(), not just the usage from cpu_sleep()? Comment Actions Taking a quick look, I do not see myself attempting freeze_timebase support for any platforms that I do not have around to test. I do not have much around compared to the variety that exist. Even for covering freeze_timebase and its use for just the 3 types of 2-socket PowerMacs that I (sometimes) have access to, it seems more likely that I'd first cover not having free_timebase support (the fallback). Then, one by one, I'd add a type-specific freeze_timebase and its use for one type of PowerMac context that I have around (at the time): a sequence of updates instead of one with everything. (One point of my general "fallback" technique is that the math for contexts with end round trip time == start round trip time effectively copies the bsp time reported to the ap over to the ap unchanged. A simple copy operation for contexts known to be that way would be an optimization instead of a fundamental change.) A question is if the fallback code should be for everything missing freeze_timebase support vs. closer to my limited range of test contexts. If "everything", then I think testing temporary variant(s) of the existing code with the alternate_timebase_sync_style=1 example(s) placed to allow the test context(s) should happen first. There is not much point in a rewrite effort that uses the fallback if the fallback technique proves insufficient to avoid the (temporarily) stuck sleeping issue as generally as required --or is problematical in some other way for contexts that I do not have access to examples of. More likely this specific submittal would be abandoned for such problems, pending an alternate fallback technique being found. Comment Actions FYI: Both of the "PowerMac11,2 with 2 sockets, 2 cores each" have died of cooling system failures. (I can sometimes boot one, but it is problematical to do much with without overheating.) So, now the only G5 I now have access to for general testing is a "2 socket/1-core-each PowerMac7,2" . I still have access to the two PowerMac3,6 with "2 sockets, 1 core each" G4 examples. (But there are other major FreeBSD kernel issues for 32-bit PowerMacs, even for single socket ones: it gradually clears some user-process pages to zero.) Comment Actions I no longer have access to any operational old-PowerMacs. (And never had access to any other example types of PowerPC/Power hardware.) As such, I'll no longer be involved in powerpc64 or 32-bit powerpc activities. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||