In D10682#222312, @andrew wrote:

In D10682#222298, @zbb wrote:

Also if we use platform_late_init there would be a need to rework all late_init implementations so that they could be used in the init_secondary.

It's not very difficult to rework platform_late_init, see D10733.

In D10682#222184, @meloun-miracle-cz wrote:

I think that this patch is not longer needed?
Both issues (device memory class remap, ACTRL modification) can be processed in platform_late_init().
I'm right?

In D9864#222185, @meloun-miracle-cz wrote:

I would prefer to move this code to platform_late_init().
The only drawback is that you must use cp15_actlr_set() + cp15_actlr_get() for boot CPU...

In D10218#221808, @skra wrote:

In D10218#221730, @zbb wrote:

@skra Why are we not able to pmap_set_tex() while caches are enabled? We are merely modifying two registers. TLB invalidation should be the only thing that is required to be done. Could you elaborate this constraint a little? It seems that this is the only thing that is holding us back from the acceptable solution so I would like to understand the issue.
Thanks in advance for your help.

My note was about two aspects of pmap_remap_vm_attr().

(1) Generally, when cache is enabled, changing cache mode of memory in active use (i.e. any memory which is mapped (keep in mind speculative reads)) is a tricky thing if possible at all. It's a problem of atomicity of such action. It's also a thing of how cache itself and cache maintainance functions work. Imagine that a page is remapped from cached mode to uncached one. All cachelines associated with this page must be removed from cache. To do this while the page is mapped as cached is not enough because of speculative reads. To do this after the page is remapped will not work as the mapping is uncached and cache maintainance function may do nothing for uncached memory. Note that access to uncached memory may still go thru cache.

Some cache modes may be change to others without problem, but it's not true for every combination of cache modes. Maybe, under very strict circumstances, it could be possible to change cache mode in some memory region.

However, pmap_remap_vm_attr() works globally for every TEX class combination. It touches all memory mapped with TEXs being changed. So, my concern was to comment such function properly to specify when it's safe to use it.

(2) Code consistency. Either some explanation should have been done why pmap_set_tex() can be used in pmap_remap_vm_attr() or comment in pmap_set_tex() should have been changed.

Well, it looks that r318251 answered my note.

@skra Why are we not able to pmap_set_tex() while caches are enabled? We are merely modifying two registers. TLB invalidation should be the only thing that is required to be done. Could you elaborate this constraint a little? It seems that this is the only thing that is holding us back from the acceptable solution so I would like to understand the issue.
Thanks in advance for your help.

In D10218#221634, @skra wrote:

In D10218#221615, @zbb wrote:

In D10218#221537, @skra wrote:

Note that pmap_remap_vm_attr() calls pmap_set_tex() which assumes that all caches are disabled (see last two lines in this function). Caches are enabled in reinit_mmu() which is called from pmap_bootstrap_prepare() and init_secondary(). It's unfortunate that Michal did not add some comment about this to pmap_remap_vm_attr(). However, it's just a note for now as use of platform_cpu_init() is not clear so far.

Yes. This must be called where it is being called now. Unfortunately platform_ code seems to be unusable for our case.

BTW. What is unclear for you regarding the use of platform_cpu_init()? There was a need to modify the default, hard-coded settings of the mappings as well as CPUs auxiliary control register. Use of another platform callback was suggested instead of ugly ifdef or another hard-coded setting of ACTLR (that was undesirable on some Cortex revisions). That is why we created this function and we use it in the related patches (please see dependencies) to modify default CPU/MMU settings.

Just wanted to say that pmap_remap_vm_attr() must be called before pmap_bootstrap_prepare(). And even if platform cpu_init() method is introduced in another place (D10682) where platform_cpu_init() is called before pmap_bootstrap_prepare(), it's used before platform code is initialized and it's wrong. So, it's unclear for now if this patch will satisfy what was noted by me.

Let us summarize what we deal with.

In D10218#221537, @skra wrote:

Note that pmap_remap_vm_attr() calls pmap_set_tex() which assumes that all caches are disabled (see last two lines in this function). Caches are enabled in reinit_mmu() which is called from pmap_bootstrap_prepare() and init_secondary(). It's unfortunate that Michal did not add some comment about this to pmap_remap_vm_attr(). However, it's just a note for now as use of platform_cpu_init() is not clear so far.

Reworked to use platform call.

In D10218#220936, @meloun-miracle-cz wrote:

Committed as r318021. Sorry for huge delay...

Reworked to use a platform call instead of a hard-coded value.

@meloun-miracle-cz your solution presented here https://github.com/strejda/tegra/commit/3b5138751ee5643992b20fcb21b280fab433bb20 looks very good.
It would be nice if you could commit this to the HEAD.

@andrew Do you have any objections to committing this patch? If not, I would like to push it to head. It was marinated long enough here :)

In D9864#205105, @meloun-miracle-cz wrote:

Ahh, I see now, Marvell subtree is not fully converted to platform model :) . But trust me, conversion its not that hard and makes your life much easier.
Also, there is global effort to make single generic kernel for all supported armv6 boards, and #ifdef SoC only technique is in direct direct contradiction with this goal.
See allwinner subtree, mainly aw_machdep.c. This platform supports numerous board with single kernel in 'right' way, without #ifdef hell.

But anyway, because SYSINIT() based proposal doesn't 'collide' with other boards/platforms, I have no objections. Now platform method seems more flexible, but that's just my personal opinion.

In D9864#204949, @meloun-miracle-cz wrote:

In D9864#204948, @zbb wrote:
In general, I don't insist on adding this change to all Cortex-A9 CPUs and I acknowledge your arguments.
Another platform-dependent method is OK for us. We need something that will be executed on both primary and secondary CPUs. It would be best if this stays in a common place for both primary and secondary CPUs (such as cpu_setup() is common for CPU0 and CPUx).
What do you think?

Yep, I have no problem with this {if cpu_setup() is new platform method). I prefer to call this new function in the place where cpuinfo is initialized and printf is working.

In D9864#204946, @meloun-miracle-cz wrote:

I'm sorry about my platform_mp_start_ap() mistake.
I understand that cpuinfo_get_actlr_modifier() looks like most direct (and most easier) way how to make this. But again, this setting must be platform dependent.
Assume that you have 2 different boards (with different bootloaders) but with exactly same CPU. One board start OS in secure world, one in non-secure. . Then, write to ACTRL is OK for first one, but may cause exception on second.

IMO, we can add another platform method, say platform_late_init_ap(), called from here https://svnweb.freebsd.org/base/head/sys/arm/arm/mp_machdep.c?view=markup#l182.

Is this sufficient for you ?

BTW. How do you expect u-boot to turn this bit on if it always works in UP?

In D9863#204631, @andrew wrote:

The BGX interface is unconnected in the ThunderX units in the netperf cluster

Any comments or test results?

In D9864#204162, @meloun-miracle-cz wrote:

Yep, fully agree. From my point of view, platform_late_init() (for BP) is best place for performance tweaks. At this point, printf() works and cpuinfo is initialized, so you can check proper CPU revision, print some warning about potentially dangerous action, etc...
For AP, you can use standard platform_mp_start_ap(), but many ACTRL bits are shared between CPU's. So, typically, it's sufficient to set up only BP.

Unfortunately, platform_late_init() is called to late for majority of errata fixups - in this case, i think that only real safe solution is bootloader fix.

@meloun-miracle-cz just to consider all possibillities. Do you have any idea whatsoever how this change could be inserted into the OS (in case there is no possibility to modify firmware), even in form of an option or platform-specific stuff?

In D9864#203677, @meloun-miracle-cz wrote:

See Erratum 571620, 719331, 719332, 751473 and probably more others.

Right place for proper CPU setup (including recommended errata fixes) is bootloader, not a boot stage of OS.

Please take in account:

• on many boards, OS starts in in non-secure mode with disabled access to ACTRL (Pandaboard is one example).
• effect of NS write to protected ACTRL is not consistent across for Cortex family, some CPU's ignore writes, some generates exception (A8, and probably some older A9).
• the current code writes to ACTRL only if bits absolutely necessary for FreeBSD run are not set. In this case, it's irrelevant if we crashes here, or few instruction later.

I think we should change this commit after all. I would prefer not to see soft_reset_del uninitialized for other drivers than Armada AHCI even though it is a part of ahci_controller structure. The better solution would be to create another quirk and activate it in the Armada AHCI wrapper according to the FDT property. We can then hard code the delay value in the generic AHCI driver and call it when the quirk is active.

This is a useful change. +1