A couple of comments:

• the firmware blobs should really live under contrib/rtlbtfw rather than being hidden away like this; kernel firmware blobs all live under sys/contrib, by contrast
• it's not clear where these blobs actually came from--could you please add a text file which describes where they come from and how they can be updated?

Rick is much more qualified to review this than me. I'm happy to go through the implementation if the idea seems sound.

I reproduced the problem as well. The issue arises when we need to bootstrap a compiler in order to do the build because the host toolchain is too old, i.e., these messages get printed:

This feature won't work with unprivileged bhyve, which can't change the owner and group of the file.

Fix the qualifier on loader_hdl.

Also, serialise each port's virtqueues with a spin mutex
and keep the transmit buffer in the per-port softc where
it outlives both the call and the port, so a system
console can transmit where the tty mutex cannot be taken.

Is there any reason to use IFT_TUNNEL in if_gre?

Fix an error path in sndbuf_remalloc().

Remove the CHANNEL_SETFRAGMENTS implementation in uaudio. It
was just returning 1 anyway, forcing a fallback to
CHANNEL_SETBLOCKSIZE.

Address review comments

Hum, I just realized that there are a few tests which require that ctld not be running, see usr.sbin/ctladm/tests/port.sh. I guess there's not much we can do about that unless we go back to requiring each test start/stop ctld, which in turn makes them slow and non-parallelizable.

I can't reproduce the problem described in the change description. I had no problems building and booting a riscv VM on an arm64 system running commit d9e2caccfac1. I ran:

Shouldn't this also get a man page update?

Looks fine to me aside from a couple of nits. Please go ahead and commit without re-uploading if you want.

In D60271#1385821, @alc wrote:

In D60271#1384940, @markj wrote:

In D60271#1383119, @alc wrote:

Yes. I would test GENERIC-NODEBUG kernels without and with this patch. I have /etc/src.conf:

WITHOUT_LLVM_ASSERTIONS=yes
#
WITH_CLEAN=yes
WITH_MALLOC_PRODUCTION=yes

I run:

#!/bin/csh
cd /usr/src
rm -fr /usr/obj/usr/src
while ( 1 )
    date
    sysctl vm.pmap vm.reserv vm.stats.vm.v_vm_faults
    time make -j16 buildworld > & /dev/null 
    rm -fr /usr/obj/usr/src
end

with -j adjusted for the machine and the script's output redirected to a log.

Unpatched output: https://reviews.freebsd.org/P717
Patched output: https://reviews.freebsd.org/P718

Timings from an unpatched kernel:

936.90 real     45789.09 user      3280.64 sys
933.97 real     45789.59 user      3314.96 sys
935.05 real     45774.25 user      3334.41 sys
934.40 real     45794.90 user      3302.60 sys
932.76 real     45846.53 user      3315.03 sys
933.13 real     45788.78 user      3318.50 sys

and a patched kernel:

938.43 real     45847.32 user      3243.15 sys
933.22 real     45670.20 user      3268.07 sys
933.84 real     45820.26 user      3269.70 sys
933.28 real     45730.27 user      3252.91 sys
939.21 real     45792.15 user      3287.05 sys
930.78 real     45835.46 user      3260.92 sys

which indicates that the change slightly reduced system CPU time.

Thanks. This system is ZFS-based, yes? Interestingly, you're getting 25% more L3C mappings than I see on UFS-based systems, presumably from loading executable files.

In D60417#1385920, @markj wrote:

In D60417#1385873, @alc wrote:

pmap_enter() should similarly be using a local invalidation when it is upgrading a R/O mapping to R/W. Comparing the amd64 and arm64 pmap_enter()s is what led me to discover all of this.

amd64 will do a (targeted) remote invalidation when upgrading a R/O mapping, I believe. Do you mean that it too should only issue a local invalidation in that case?

In D60417#1385873, @alc wrote:

pmap_enter() should similarly be using a local invalidation when it is upgrading a R/O mapping to R/W. Comparing the amd64 and arm64 pmap_enter()s is what led me to discover all of this.

Do you have any script which could be used to verify the linker script after future imports?

In D60384#1385022, @jtranoleary_google.com wrote:

This patch and the two that follow it fix boot panics caused by gve on some ARM images which don't have https://reviews.freebsd.org/D59476 . gve fails to attach properly due to DMA issues and then panics when trying to unwind allocations due to null pointer dereferences.

Where is the include loop exactly? I can't see it. IMO we should try to fix that instead of introducing a new header.

Different cloner drivers are quite inconsistent with each other about which sysinit subsystem they belong to. Some further cleanup here would be great, here I am just trying to minimally fix a panic I see when running the test suite with the linuxulator loaded.

In D60271#1383119, @alc wrote:

Yes. I would test GENERIC-NODEBUG kernels without and with this patch. I have /etc/src.conf:

WITHOUT_LLVM_ASSERTIONS=yes
#
WITH_CLEAN=yes
WITH_MALLOC_PRODUCTION=yes

I run:

#!/bin/csh
cd /usr/src
rm -fr /usr/obj/usr/src
while ( 1 )
    date
    sysctl vm.pmap vm.reserv vm.stats.vm.v_vm_faults
    time make -j16 buildworld > & /dev/null 
    rm -fr /usr/obj/usr/src
end

with -j adjusted for the machine and the script's output redirected to a log.

In D60271#1383044, @markj wrote:

I can test this patch on it this weekend if you tell me what exactly you'd like to try.

In D40022#1384219, @imp wrote:

So I don't know if this landed in main, or a better one did, but I'm going to walk away from this one.

In D60271#1383042, @alc wrote:

Do any of you have access to a machine with IDC, but not DIC, e.g., an older Ampere I think? The results on a small Cortex-X1/A78 system are at best inclusive.