Hyperv: vmbus: Use INTR_MPSAFE in swi_add and bind swi thread to each CPU.

[honzhan_microsoft.com]

Summary

Software interrupt thread for msg and event handler should be binded to each CPU,
and since it is not required to sync between those threads, INTR_MPSAFE is the
correct flag.

Submitted by: Hongjiang Zhang <honzhan microsoft com>

[howard0su_gmail.com]

sys/dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c

524–525

I suggest we remove this per-CPU swi since message is not performance critical.

[decui_microsoft.com]

sys/dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c

524–525

I discussed this with Jun and we found we have to keep the per-cpu swi messaage thread: the message must to be handled on the cpu where the interrupt is received and we can only make sure this by keeping the per-cpu swi thread.

BTW, actually the "intr_event_bind(hv_vmbus_g_context.hv_msg_intr_event[j], j)" is unnecessary, however we can leave it as it is.

BTW2, actually there is no need of INTR_MPSAFE for the msg swi thread, because the (non-timer) messages can only be delivered to one CPU, i.e., normally it's CPU0 only.

[howard0su_gmail.com]

sys/dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c

524–525

the message only delivers to one CPU doesn't mean we don't need INTR_MPSAFE. it means we can give INTR_MPSAFE without doing any locking inside the swi handler. Grab Gaint when running swi handler can cause other lock contention that need Gaint lock in other subsystem other than the swi handler in hyperv driver.

[royger]

LGTM.

[decui_microsoft.com]

LGTM

sys/dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c

524–525

I agree.

[howard0su_gmail.com]

I find another way to remove SWI. We can add "struct task" for each CPU, and schedule a task in hard interrupt handler via taskqueue_enqueue(taskqueue_fast, task[curcpu]). Then we don't need spawn a thread for each CPU. In such way, the code logic is actual more simple.

[sepherosa_gmail.com]

In D4800#105197, @howard0su_gmail.com wrote:

I find another way to remove SWI. We can add "struct task" for each CPU, and schedule a task in hard interrupt handler via taskqueue_enqueue(taskqueue_fast, task[curcpu]). Then we don't need spawn a thread for each CPU. In such way, the code logic is actual more simple.

Using per-cpu taskq thread is used to avoid 'remote schedule', i.e. ipi.

By reusing taskqueue_fast, you end up w/ only one event handler thread; that's obviously not what we want for the upcoming vRSS support.

[howard0su_gmail.com]

In D4800#105199, @sepherosa_gmail.com wrote:

In D4800#105197, @howard0su_gmail.com wrote:

I find another way to remove SWI. We can add "struct task" for each CPU, and schedule a task in hard interrupt handler via taskqueue_enqueue(taskqueue_fast, task[curcpu]). Then we don't need spawn a thread for each CPU. In such way, the code logic is actual more simple.

Using per-cpu taskq thread is used to avoid 'remote schedule', i.e. ipi.

By reusing taskqueue_fast, you end up w/ only one event handler thread; that's obviously not what we want for the upcoming vRSS support.

The change is for message which is mainly the commands to setup channel, remove channel, etc. they are not performance critical like event handling. Or they are not related to NIC performance.

[sepherosa_gmail.com]

In D4800#105200, @howard0su_gmail.com wrote:

In D4800#105199, @sepherosa_gmail.com wrote:

In D4800#105197, @howard0su_gmail.com wrote:

I find another way to remove SWI. We can add "struct task" for each CPU, and schedule a task in hard interrupt handler via taskqueue_enqueue(taskqueue_fast, task[curcpu]). Then we don't need spawn a thread for each CPU. In such way, the code logic is actual more simple.

Using per-cpu taskq thread is used to avoid 'remote schedule', i.e. ipi.

By reusing taskqueue_fast, you end up w/ only one event handler thread; that's obviously not what we want for the upcoming vRSS support.

The change is for message which is mainly the commands to setup channel, remove channel, etc. they are not performance critical like event handling. Or they are not related to NIC performance.

OK, I thought you meant to use taskq_fast for both events and msgs.

[honzhan_microsoft.com]

• hyperv: vmbus: refactor msg handler by replacing swi with task queue