The changes looks good.

There us a similar problem when doing tunneling. For instance, I have to use a gif tunnel that encapsulates IPv6 packets inside IPv4 packets that are sent to a remote 6rd gateway in order to get IPv6 connectivity to the outside world. It would be nice to have the option of peering into these encapsulated packets to see the individual flows. Care would have to be taken to make sure that we don't walk off the end of the mbuf when doing this.

Ping

Incorporate trysteal improvement from Jeff Roberson @ Isilon

In D12130#254431, @mav wrote:

At high context switching rates on systems with many cores stealing code does create significant CPU load. So I can believe that the critical section can indeed be an issue, so this way may be good to go. I personally thought about limiting maximal stealing distance based on some statistic factors, such as context switch rates, etc, since we shouldn't probably touch every CPU caches on every context switch -- its a dead end with growing number of CPUs, but haven't got far on it.

Builds fine and doesn't appear to break anything on my Ryzen machine.

Builds fine and doesn't cause any obvious breakage on my Ryzen. This feature does appear to be detected properly.

Any feedback? I'd like to get this committed before I RMA my Ryzen CPU.

In D12130#252053, @kib wrote:

In D12130#252045, @truckman wrote:

I don't think that there is a problem with ithread migration. As a matter of fact the lack of any obvious problems with ithreads makes me suspect a tlb issue. Ithreads live in kernel memory space which is going to be the same everywhere. I'm wondering if a thread that has migrated to a core on the other CCX isn't always getting its tlb fully initialized before returning to userspace.

We do not re-initialize (flush is the common term) thread's TLB before returning to userspace. We flush TLB when switching contexts. More precisely, there are two variants of it.

On older CPUs, without the PCID feature, reload of %cr3 (the page table base pointer) flushes all non-global entries from TLB of the CPU thread which reloaded %cr3. Kernel-mode TLB entries are typically global (PG_G). The reload most often occurs on context switch, see cpu_switch.S.

On newer Intel CPUs, where PCID feature is present, each TLB entry is additionally tagged with the address space ID, and on the switch we typically inform CPU about new address space ID, avoiding the flush.

One of the reason I asked you about the verbose dmesg some time ago was to see which TLB switching mechanism is used, and apparently Ryzens do not implement PCID. Although AMD CPUs do tag TLB entries with ASIDs for SVM for long time.

You could experiment by adding the the equivalent of invltlb_glob() right after %cr3 reload in cpu_switch. This should flush the whole TLB, including the kernel entries.

You already tried to disable superpage promotions, so other known workarounds like erratum383 should be not useful.

Mark more members of struct tdq as volatile to prevent accesses
to them from being optimized out by the compiler. They are
written in one context and read in another, sometimes multiple
times in one function, and we want the reader to notice changes
if they happen.

In D12130#253585, @cem wrote:

In D12130#253523, @avg wrote:

In D12130#251884, @jeff wrote:

I have another patch that was a 2-3% perf improvement at isilon that I have been meaning to backport that is somewhat at odds with this, although not entirely. It performs a search in the last thread to switch out rather than switching to the idle thread. This saves you a context switch into the idle thread and then back out again when stealing will be immediately productive. The code is relatively straightforward. It helps a lot when there are tons of context switches and lots of short running threads coming and going.

I would love to see that change land in the tree. It makes a lot of sense to elide a switch to the idle thread if we have an opportunity to steal a thread and switch to it directly.

https://people.freebsd.org/~cem/0001-Improve-scheduler-performance-and-decision-making.-F.patch if you want to work on adopting it. I've committed the standalone subr_smp.c portion already.

In D12130#253523, @avg wrote:

In D12130#251884, @jeff wrote:

I have another patch that was a 2-3% perf improvement at isilon that I have been meaning to backport that is somewhat at odds with this, although not entirely. It performs a search in the last thread to switch out rather than switching to the idle thread. This saves you a context switch into the idle thread and then back out again when stealing will be immediately productive. The code is relatively straightforward. It helps a lot when there are tons of context switches and lots of short running threads coming and going.

I would love to see that change land in the tree. It makes a lot of sense to elide a switch to the idle thread if we have an opportunity to steal a thread and switch to it directly.

In D12217#253500, @rozhuk.im-gmail.com wrote:

Works on my Ryzen machine, so I'm no longer flying blind in terms of CPU temperature under load. Doing mental math or manually setting the temperature offset is fine for now.

Should the 20C temperature offset for the 1700X and 1800X be set automatically? When I first got my 1700X, the motherboard had the original BIOS which did not account for the offset. It thought the CPU temperature was 50+C at idle and the fan ran at close to full speed even though I've got a massive heat sink on the CPU. A subsequent BIOS upgrade compensated for the offset and told me that the idle temperature was in the low 30C range and the fan speed was much more reasonable at idle.

I did some further investigation of the successful steals that happen when steal->tdq_load == 1 && steal->tdq_transferrable !=0. That happened 1557 times during a buildworld run. Of those, steal->tdq_cpu_idle was nonzero 1099 times, and steal->tdq_ipipending was nonzero 196 times. It doesn't look like the source CPU being busy handling an interrupt was the issue. On my hardware, cpu 0 handles the majority of interrupts by far, but the distribution these events is fairly evenly distributed across all of the CPUs.

As a test case, you might try building lang/ghc in parallel with some other load so that there is some thread migration activity. I'm not sure it is the same problem, but ghc seems to be super sensitive and I pretty much always see SIGBUS failures on my Ryzen machine.

AMD has publicly been very vague about this problem. About the only thing they've said is that there is some sort of "performance marginality" and that Threadripper is not affected. Since Threadripper uses the same die stepping as Ryzen, it looks like they have figured out how to screen for parts with this problem. There is a long thread on the AMD forum here: https://community.amd.com/thread/215773?start=1095&tstart=0.

Basically this started as an investigation of system hangs, reboots, and oddball process failures on Ryzen as documented in this PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219399. It turns out that there are actually multiple issues and this PR ended up tracking the hang/reboot issue, which was eventually fixed. The remaining problem(s) were moved to this PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221029. After doing a number of experiments, I came to the conclusion that the majority of the problems were correlated to thread migration, and I discovered that disabling both kern.sched.balance and kern.sched.steal_idle got rid most or all of the problems. That led me to start hacking on the scheduler code to see if there was any pattern to the problematic thread migrations and I found that disabling the last loop iteration that looked at all cores alleviated the problems. Ryzen apparently has issues with its IRET instruction, so I wasn't sure if the problem was triggered by having interrupts disabled for such a long time or if the problem was due to migrating threads between the two CCXes. It also looked to me like there was an off-by-one bug in sched_highest(), so I asked about that in this email thread: https://docs.freebsd.org/cgi/getmsg.cgi?fetch=12902+0+/usr/local/www/mailindex/archive/2017/freebsd-arch/20170827.freebsd-arch (I don't know why my original message isn't in the archive).

Lower sv_maxuser as well to prevent a user from mapping and loading
code into the page at 0x7ffffffff000. Executing code there can cause
the system to hang or silently reboot.

I just tried an experiment where I mapped the page at 0x7ffffffff000 and loaded it with some trivial code. I was able to execute that code once just fine, but if I loop on it so that the process spends most of its time executing code from that page, I get a nearly instant reboot.

In D11780#244444, @ed wrote:

Don, thanks for pointing me to this discussion!

In our case shared pages tend to reside at those addresses, but doesn't this problem apply to simply any mapping there? As in, even if you would lower the mapping of the shared page, people could still use mmap() to place a page at the very top and exploit this issue, right? In other words, shouldn't we lower VM_MAXUSER_ADDRESS instead?

Right now VM_MAXUSER_ADDRESS is assumed to be a constant, so lowering that is going to be annoying. That said, is it really worth the trouble distinguishing its value based on the CPU that's currently in use? I personally wouldn't mind to lose a single page of virtual memory.

Update patch for linux change r321728.

cloudabi doesn't appear to use the shared page.

Add the amd64_lower_shared_page() prototype to md_var.h.

Renamed elf64_freebsd_sysentvec_fixup() to amd64_lower_shared_page(),
changed it to use decrements, and reused it in the Linux code. Where
should its prototype be declared?

I can't say that I'm enthusiastic about putting the fixup function in initcpu.c and polluting it with <sys/syent.h> stuff. All of the other code in this file pretty much sticks to poking at the CPU itself.

One thing that I like about the way the tunable works is that the mode is auto if it is unset by the loader. Setting it in the loader forces the chosen mode. The sysctl value reflects the actual mode chosen by the combination of the tunable and the cpuid test.

About my inline comment,the patch was working, but I was really puzzled by why the trampoline address was changing by more than PAGE_SIZE. Even when I used the tunable to disable lowering of the shared page (and even commented out the code that did the adjustment), the lower bits of the trampoline address were going from 0x190 to 0x000. The behavior depended on whether or not the new SYSINIT was present in the code. It took me quite a while, but I eventually figured out that adding the new SYSINIT was perturbing the ordering of the other SYSINITs. In the original code, the 32-bit sysvec was getting initialized first, so the 32-bit stuff would get loaded into the start of the shared page. When a new SYSINIT is added, the 64-bit sysvec is initialized first and the 64-bit trampoline is the first item in the shared page.

The IS_BSP() check isn't sufficient because that code fragment also gets executed on resume and everything would get decremented again.

I just got a failure with this patch on 12.0-CURRENT r318776 amd64.

On

CPU: AMD Ryzen 7 1700X Eight-Core Processor          (3393.69-MHz K8-class CPU)

I get the following results:

sysctl dev.amdtemp

dev.amdtemp.0.rtc.sensor_offset: 0
dev.amdtemp.0.rtc.PerStepTimeUp: 0
dev.amdtemp.0.rtc.PerStepTimeDn: 0
dev.amdtemp.0.rtc.TmpMaxDiffUp: 0
dev.amdtemp.0.rtc.TmpSlewDnEn: 0
dev.amdtemp.0.rtc.CurTmpTjSel: -49.0C
dev.amdtemp.0.rtc.CurTmp: 0.1C
dev.amdtemp.0.%parent: hostb10
dev.amdtemp.0.%pnpinfo:
dev.amdtemp.0.%location:
dev.amdtemp.0.%driver: amdtemp
dev.amdtemp.0.%desc: AMD CPU On-Die Thermal Sensors
dev.amdtemp.%parent: