vmm: Fix HLT loop while vcpu has requested virtual interrupts

[gusev.vitaliy_gmail.com]

Summary

We have seen stuck of Linux VMs (kernel 5.4, 5.15, 6.x) with messages in Linux's dmesg:

1. BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 37s!
2. watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:3:2090493]
3. rcu: INFO: rcu_sched detected stalls on CPUs/tasks:

Physical CPUs are Intel(R) Xeon(R) Gold 6248R and have enabled "Virtual Interrupt Delivery" and "Process posted interrupts" features.

Investigation shown that vmx_pending_intr returns 0 even if vmexit->u.hlt.intr_status was 0xfe, 0xfd, ... and per_desc->pending was 1.

Function vmx_inject_pir has mention about situation when 'pending' is 1 and zero pirval-s:

* It is possible for pirval to be 0 here, even though the                                                                                                          
* pending bit has been set. The scenario is:

Correct initial fix "02cc877968bbcd57695035c67114a67427f54549 Recognize a pending virtual interrupt while emulating the halt instruction" for all cases: pending is 0 and pending is 1.

Possible issue is also mentioned here: debian-vm-freezes-after-several-hours

Sponsored by: vStack

Test Plan

Check several types of VM: Linux, Windows, FreeBSD.

Verify that VM has idle time, verify that Linux VMs (RHEL9, OEL9, etc.) don't have lockups and RCU stalls anymore.

[corvink]

LGTM

[gusev.vitaliy_gmail.com]

@jhb @markj Could you look at it? This fix potentially could be added to 13.2 .

[gusev.vitaliy_gmail.com]

up?

[markj]

What exactly is the problematic scenario? How do we end up with a pending, undelivered interrupt after a vmexit? Presumably the guest must have enabled interrupts before executing HLT.

The change itself seems reasonable to me, but I'd like to understand it better.

sys/amd64/vmm/intel/vmx.c

3785

This is section 30.2.1 now.

[gusev.vitaliy_gmail.com]

In D39620#905476, @markj wrote:

What exactly is the problematic scenario? How do we end up with a pending, undelivered interrupt after a vmexit? Presumably the guest must have enabled interrupts before executing HLT.

The change itself seems reasonable to me, but I'd like to understand it better.

Look at the vmx.c L4016
It has good explanation how it can occur. One more example could be w/o involving posted-interrupt due to async changes between pending and pir[]

Interrupt can be undelivered also due to not true conditions in "evaluation of pending interrupt" 30.2 of SDM.

Presumably the guest must have enabled interrupts before executing HLT.

It has already check on it. Look at vm_handle_hlt(). Problem is not in disabled interrupts during HLT. It is enabled.

[gusev.vitaliy_gmail.com]

Correct mention of SDM : Section 29.2.1 --> Section 30.2.1

[sean_rogue-research.com]

Hi all,

I believe I am experiencing this bug on TrueNAS 13.0-U5.2 (not quite the newest, but almost).

I'm not very familiar with phabricator, so I'm unsure: is this fixed in FreeBSD 13.2 or only 14.0?

Is there any workaround until TrueNAS is updated to a newer FreeBSD?

Many thanks.

[gusev.vitaliy_gmail.com]

In D39620#978503, @sean_rogue-research.com wrote:

...
I'm not very familiar with phabricator, so I'm unsure: is this fixed in FreeBSD 13.2 or only 14.0?

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

Is there any workaround until TrueNAS is updated to a newer FreeBSD?

You have reproducers in Linux, right ? Probably you could try 6.x Linux kernel. Please report if it helps.

[sean_rogue-research.com]

stable/13 has this patch

OK great. TrueNAS will probably go there before they go to 14.

You have reproducers in Linux, right ? Probably you could try 6.x Linux kernel. Please report if it helps.

Assuming the bug I see is indeed this bug, I reproduce in all 3 of my linux VMs, running Ubuntu 20.04 LTS and 22.04 LTS, which are all linux kernel 5.x. Looks like there is no LTS Ubuntu with kernel 6.x until upcoming 24.04. But 23.10 uses kernel 6.5, which I could try. Is kernel 6.x expected to fix something?

[gusev.vitaliy_gmail.com]

In D39620#978525, @sean_rogue-research.com wrote:

Assuming the bug I see is indeed this bug, I reproduce in all 3 of my linux VMs, running Ubuntu 20.04 LTS and 22.04 LTS, which are all linux kernel 5.x. Looks like there is no LTS Ubuntu with kernel 6.x until upcoming 24.04. But 23.10 uses kernel 6.5, which I could try. Is kernel 6.x expected to fix something?

It could be, but I would expect that Linux 6.x kernel just changed behaviour a little and therefore race caused that issue is rarely seen.

[sean_rogue-research.com]

It could be, but I would expect that Linux 6.x kernel just changed behaviour a little and therefore race caused that issue is rarely seen.

OK, we reproduce easily with an Ubuntu 22.04.6 VM running gitlab. It freezes every few days. I'll try to update the server to 23.10 and see what happens.

And just to link together a few things, in case it helps others (we've been chasing this bug for months, part time):

• Related TrueNAS ticket: https://ixsystems.atlassian.net/browse/NAS-122108
• Similar old FreeBSD ticket: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222916
• Possibly related: https://reviews.freebsd.org/rG2c352feb3bf93c679f9e41d65bc8dc8394a4fdae
• Possibly related: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275155
• TrueNAS forum thread 1: https://www.truenas.com/community/threads/debian-vm-freezes-after-several-hours.106367/#post-742515
• TrueNAS forum thread 2: https://www.truenas.com/community/threads/bhyve-with-ubuntu-19-04-keeps-locking-up.79191/page-3#post-757796

[sean_rogue-research.com]

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

I'm not very familiar with FreeBSD's branching system... I see FreeBSD 13.3-RELEASE was released today, is this bug fix included?

[mav]

In D39620#1008905, @sean_rogue-research.com wrote:

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

I'm not very familiar with FreeBSD's branching system... I see FreeBSD 13.3-RELEASE was released today, is this bug fix included?

Yes. releng/13.3 was branched from stable/13 just about a month ago, so it sure got it.