Add ThunderX as FreeBSD target

[wma_semihalf.com]

Summary

Add Cavium ThunderX as a FreeBSD target

Provide basic file structure and kernel config to support
Cavium ThunderX armv8 hardware. From now, every one
can compile his own kernel for ThunderX system.

Features:
- support for GICv3, UART
- basic userspace operation from ramfs in single-user mode

[emaste]

sys/arm64/conf/THUNDER-88XX

3

update comment to match ident

28

prefer include GENERIC and then use options / device / nooptions / nodevice to adjust - afaict it should only be the SOC_CAVM_THUNDER for workarounds and the temporary fdt inclusion?

88–89

add a comment explaining why we have SOC_CAVM_THUNDER please
my understanding is that it should be only for early hw workarounds, and production ThunderX will be able to use GENERIC arm64

[wma_semihalf.com]

sys/arm64/conf/THUNDER-88XX

28

Agreed, that will be much cleaner. Thanks.

88–89

I'm not sure we can ever get rid of machine-specific kernconfigs. The SOC_CAVM_THUNDER is used for some tweaks in ITS (not commited yet). We also have a huge problem with PCI drivers. As far as I know, there can be only one PCI-bus driver compiled in kernel, so we need to have a method to differentiate which driver file should be added to compilation, especially when we add a support for other armv8 targets with pcie. There is also a KSTACK_PAGES and MAXCPUS which are used for calculating table sizes and I don't see any easy method to resolv them dynamically or find any common value.
This is why I suggest leaving this file as is and continue working on FreeBSD 'generalization' in parallel. Somehow I feel it requires much more effort than we anticipate at the moment...

[emaste]

sys/arm64/conf/THUNDER-88XX

88–89

Having a unified kernel and install media is a very important goal for the port, so we should try to avoid #ifdefs as much as possible. Using it for workarounds for early hardware errata are not a problem, of course. x86 has similar arguments for things like MAXCPUS and KSTACK_PAGES and manages with a single kernel config -- I see no problem making it say 128 or 256 in GENERIC, and someone compiling for small hardware can tune it down. The PCI driver question is certainly a bigger issue.

Anyhow, my point is that there's no problem relying on this option right now where we don't have an alternative, but we should try to avoid it whenever possible.

[wma_semihalf.com]

sys/arm64/conf/THUNDER-88XX

88–89

Understood. I clean up this file and leave only absolutely necessary options. I think that we'd need SOC_CAVM_THUNDER, BOOTVERBOSE (the infamous timing issue with SMP init, still not resolved) and FDT stuff, since our EFI does not provide dtb.

[wma_semihalf.com]

Moved some options which will be necessary in the future from THUNDER-88XX to GENERIC. Cleaned up Thunder kenrconf and left only absolutely necessary options.

[andrew]

Why do we need a new kernel config with a static dtb? I'm strongly opposed to this as we should be making it work with GENERIC.

[wma_semihalf.com]

We will not be able to make it generic until the issue with PCI compilation is resolved. Maybe I didn't specified it clearly enough, but in the present FreeBSD state we need to have a separate config files. If you think we can live with fbsd without any IO interfaces working on the real hardware, then we can definitely stick to GENERIC. Keep in mind, that configuring pci in GENERIC does not cope well with an acpi option and makes the compilation fail...
I'd really like to have this change pushed as an enabler for all further ThunderX support. It is only a temporary and will be removed as soon as is no longer necessary.

Removing SOC_CAVM_THUNDER is also doable if that ensures you that no mahcine-specific ifdefs are present and the integration will go smoothly.

I think I can remove dts from this patch. The latest EFI from Cavium should have it fixed, but need to check.

[bz]

sys/arm64/conf/THUNDER-88XX

26

That should not go into a kernel config; just set it in loader.con given that is supported on arm64.

sys/conf/options.arm64

6

opt_global.h is most likely not the right file to put an architecture specific, non-global option in. Going by example here seems not the right thing to do ;-)

[andrew]

You haven't listed any problems that can't be fixed in GENERIC. It would help if you could tell me the specific problems you are trying to solve and why you couldn't use, for example, quirks to solve them.

sys/arm64/conf/GENERIC

72

Why do we need this? It's only 8 on powerpc64, everywhere else it's either 2 or 4, with the latter more common.

[emaste]

sys/arm64/conf/THUNDER-88XX

26

It is a workaround for some timing or stack related issue, and in some PXE boot cases we're not picking up loader.conf. It at least needs a comment explaining why it's here.

[wma_semihalf.com]

In D3008#59529, @andrew wrote:

You haven't listed any problems that can't be fixed in GENERIC. It would help if you could tell me the specific problems you are trying to solve and why you couldn't use, for example, quirks to solve them.

An example:
Using GENERIC with "option pci" enabled makes the kernel fail to compile. There is whole bunch of pci_cfgregread* functions that are x86 specific which are not implemented on any other architecture. Right now, acpi support for arm64 is incomplete and either needs to be improved or proper wrappers are to be implemented. Either way the easier option is to define "nodevice acpi" in machine-specific kernel config, because any modification to acpi drivers could take too much time and delays us from getting ThunderX support for few months. Or to remove it from GENERIC due to issues it is causing.

[imp]

In D3008#59571, @wma_semihalf.com wrote:

In D3008#59529, @andrew wrote:

You haven't listed any problems that can't be fixed in GENERIC. It would help if you could tell me the specific problems you are trying to solve and why you couldn't use, for example, quirks to solve them.

An example:
Using GENERIC with "option pci" enabled makes the kernel fail to compile. There is whole bunch of pci_cfgregread* functions that are x86 specific which are not implemented on any other architecture. Right now, acpi support for arm64 is incomplete and either needs to be improved or proper wrappers are to be implemented. Either way the easier option is to define "nodevice acpi" in machine-specific kernel config, because any modification to acpi drivers could take too much time and delays us from getting ThunderX support for few months. Or to remove it from GENERIC due to issues it is causing.

How does MIPS have PCI bus then? Or should that first line be 'option acpi'?

I'd strongly recommend that a bug be filed for each of these issues, and reference it from the thunder kernel config file. This will document why we don't have GENERIC support for that platform and allow the greater project visibility into the issues that are known about inside of semihalf for this platform. One of the problems that we've had in the past with the non-x86 platforms is that something will be done and committed that's known to be not quite done or right, then others copy it not knowing the defects, making it really hard to clean up later.

sys/arm64/conf/THUNDER-88XX

30

Why isn't the boot loader supplying the dtb if the EFI isn't? And if the boot loader isn't working, that's a problem that should be documented so we can kill the workaround when it gets fixed.

[wma_semihalf.com]

How does MIPS have PCI bus then? Or should that first line be 'option acpi'?

Sorry, let me clarify. Using "device acpi" and "option pci" togather make a compilation fail on armv8. Acpi_pci uses some pci_cfg functions that are defined in x86 machdep files.

Why isn't the boot loader supplying the dtb if the EFI isn't? And if the boot loader isn't working, that's a problem that should be documented so we can kill the workaround when it gets fixed.

We are using TFTP/PXE to boot the board and the loader just don't work well with ThunderX-EFI network devices (EFI lacks some functionality, to be precise). Loading the kernel via TFTP is sometimes too much. Newer EFI should provide updated dtb which is identical to the one attached, but I'll check it tommorow.

I will clean up this patch a little and and comment all hacks and workarounds we're using inside kernconf. Also I will modify the header to indicate it is only a temporary solution and no one should ever copy the contents of this file into anything else than /dev/null.

[wma_semihalf.com]

sys/arm64/conf/GENERIC

72

We have found some cases where 4 was too low, so this is why we're using 8 on ThunderX.

[emaste]

In D3008#59583, @imp wrote:

I'd strongly recommend that a bug be filed for each of these issues, and reference it from the thunder kernel config file.

Agreed, this sounds like a good approach. I wanted at least a comment on each of them, but PRs for each one will be much better for tracking.

[emaste]

sys/arm64/conf/GENERIC

72

It would be useful to capture example backtraces of these in a PR

[andrew]

In D3008#59595, @wma_semihalf.com wrote:

How does MIPS have PCI bus then? Or should that first line be 'option acpi'?

Sorry, let me clarify. Using "device acpi" and "option pci" togather make a compilation fail on armv8. Acpi_pci uses some pci_cfg functions that are defined in x86 machdep files.

Then why not implement the missing functions as a stub calling panic? As you are booting using fdt this is not an issue, and can be fixed later.