Page MenuHomeFreeBSD
Differential D55492

virtio_net: Use bus_dma for rxq/txq buffers
Needs ReviewPublic
Actions

Authored by sarah.walker2_arm.com on Tue, Feb 24, 4:29 PM.
Tags
None
Referenced Files
F146494292: D55492.diff
Tue, Mar 3, 3:35 AM
Unknown Object (File)
Sun, Mar 1, 2:24 PM
Unknown Object (File)
Sun, Mar 1, 3:33 AM
Unknown Object (File)
Sun, Mar 1, 2:26 AM
Unknown Object (File)
Sun, Mar 1, 2:25 AM
Unknown Object (File)
Sat, Feb 28, 3:34 PM
Unknown Object (File)
Sat, Feb 28, 7:22 AM
Unknown Object (File)
Fri, Feb 27, 2:57 AM
View All 20 Files
Subscribers
br
imp

Details

Reviewers
bryanv
andrew
Summary

While the majority of virtio platforms will be fully coherent, some may
require cache maintenance or other specific device memory handling (eg for
secure partitioning). Using bus_dma allows for these usecases.

The virtio buffers are marked as coherent; this should ensure that sync
calls are no-ops in the common cases.

Sponsored by: Arm Ltd

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 71070
Build 67953: arc lint + arc unit

Event Timeline

sarah.walker2_arm.com created this revision.Tue, Feb 24, 4:29 PM
Herald added a subscriber: imp. · View Herald TranscriptTue, Feb 24, 4:29 PM
sarah.walker2_arm.com requested review of this revision.Tue, Feb 24, 4:29 PM
Harbormaster completed remote builds in B70961: Diff 172610.Tue, Feb 24, 4:29 PM
andrew added reviewers: bryanv, andrew.Wed, Feb 25, 10:18 AM
andrew added a subscriber: br.
br added a comment.Wed, Feb 25, 11:44 AM

A quick test on RISC-V IOMMU gives me this issue:

vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
panic: alignment failed: ctx 0xffffffd0010d0e80 start 0x20c000 offset 82 align 0x4
sarah.walker2_arm.com added a comment.Wed, Feb 25, 11:50 AM
In D55492#1270046, @br wrote:

A quick test on RISC-V IOMMU gives me this issue:

vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
panic: alignment failed: ctx 0xffffffd0010d0e80 start 0x20c000 offset 82 align 0x4

Was there a backtrace?

br added a comment.Wed, Feb 25, 11:54 AM
vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
panic: alignment failed: ctx 0xffffffd0010d0e80 start 0x20c000 offset 82 align 0x4
cpuid = 0
time = 1772019624
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_fetch_ksymtab() at db_fetch_ksymtab+0x142
kdb_backtrace() at kdb_backtrace+0x2c
vpanic() at vpanic+0x16e
panic() at panic+0x26
iommu_domain_fini() at iommu_domain_fini+0x36a
iommu_is_buswide_ctx() at iommu_is_buswide_ctx+0x806
bus_dmamap_load_mbuf_sg() at bus_dmamap_load_mbuf_sg+0x60
vtmmio_attach() at vtmmio_attach+0x488c
vtmmio_attach() at vtmmio_attach+0x6656
vtmmio_attach() at vtmmio_attach+0x5652
ether_ioctl() at ether_ioctl+0x1a0
vtmmio_attach() at vtmmio_attach+0x5844
in_control_ioctl() at in_control_ioctl+0x9ac
rtnl_ifaces_destroy() at rtnl_ifaces_destroy+0x1a1e
rtnl_register_messages() at rtnl_register_messages+0x2a0
nl_send() at nl_send+0x3ca
nl_taskqueue_handler() at nl_taskqueue_handler+0x34e
taskqueue_run() at taskqueue_run+0x1f0
taskqueue_thread_loop() at taskqueue_thread_loop+0xd4
fork_exit() at fork_exit+0x68
fork_trampoline() at fork_trampoline+0xa
KDB: enter: panic
[ thread pid 0 tid 100062 ]
Stopped at      kdb_enter+0x3a: sd      zero,560(s1)
andrew added a comment.Wed, Feb 25, 12:02 PM

Should the alignment for the rx & tx DMA tags be VTNET_ETHER_ALIGN?

br added a comment.Wed, Feb 25, 12:05 PM

note function names in backtrace sometimes wrong, so it is more like

bus_dmamap_load_mbuf_sg() -> iommu_bus_dmamap_load_buffer() -> iommu_bus_dmamap_load_something() -> iommu_bus_dmamap_load_something1() -> panic
br added a comment.Wed, Feb 25, 12:08 PM
In D55492#1270064, @andrew wrote:

Should the alignment for the rx & tx DMA tags be VTNET_ETHER_ALIGN?

works with that. There are some warnings, but ping works

vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
vtnet0: link state changed to UP
vtnet0: error setting host MAC filter table
Starting Network: lo0 vtnet0.
sarah.walker2_arm.com added a comment.Wed, Feb 25, 12:45 PM
In D55492#1270066, @br wrote:
In D55492#1270064, @andrew wrote:

Should the alignment for the rx & tx DMA tags be VTNET_ETHER_ALIGN?

works with that. There are some warnings, but ping works

vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
vtnet0: link state changed to UP
vtnet0: error setting host MAC filter table
Starting Network: lo0 vtnet0.

Do the warnings show without this change?

sarah.walker2_arm.com updated this revision to Diff 172677.Wed, Feb 25, 2:14 PM

Attempted fix for alignment issue.

Harbormaster completed remote builds in B70991: Diff 172677.Wed, Feb 25, 2:14 PM
sarah.walker2_arm.com added a comment.Wed, Feb 25, 2:15 PM
In D55492#1270046, @br wrote:

A quick test on RISC-V IOMMU gives me this issue:

vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
panic: alignment failed: ctx 0xffffffd0010d0e80 start 0x20c000 offset 82 align 0x4

Could you try again with the most recent change please?

br added a comment.Wed, Feb 25, 2:20 PM

Could you try again with the most recent change please?

same panic

sarah.walker2_arm.com updated this revision to Diff 172679.Wed, Feb 25, 2:33 PM

Actually fix alignment issue.

sarah.walker2_arm.com added a comment.Wed, Feb 25, 2:33 PM
In D55492#1270135, @br wrote:

Could you try again with the most recent change please?

same panic

Apologies, not sure what I was thinking with the previous change. Can you try the current change please?

Harbormaster completed remote builds in B70992: Diff 172679.Wed, Feb 25, 2:34 PM
br added a comment.Wed, Feb 25, 2:42 PM
In D55492#1270142, @sarah.walker2_arm.com wrote:
In D55492#1270135, @br wrote:

Could you try again with the most recent change please?

same panic

Apologies, not sure what I was thinking with the previous change. Can you try the current change please?

It works but shows this warnings. Also vtnet does not seem to work in iommu bypass mode (hangs after "lo0: link state changed to UP").
Could be riscv iommu issue. I'll check on my side. vtblk works fine in all cases

lo0: link state changed to UP
vtnet0: unable to set MAC address
vtnet0: cannot disable promiscuous mode
vtnet0: cannot disable all-multicast mode
vtnet0: error setting host MAC filter table
vtnet0: link state changed to UP
vtnet0: error setting host MAC filter table
Starting Network: lo0 vtnet0.
br added a comment.Wed, Feb 25, 5:31 PM

looks like some part of vtnet still tries to pass physical addresses instead of bus dma addresses, so I get page faults

riscv_fq_intr
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940a iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940a iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940d iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940d iotval2: 0x0
riscv_iommu0: Fault: event 0xf received: Write/AMO page fault
riscv_iommu0: iotval 0x80e4940f iotval2: 0x0
riscv_iommu0: Fault: event 0xf received: Write/AMO page fault
riscv_iommu0: iotval 0x80e4940f iotval2: 0x0
vtnet0: cannot disable all-multicast mode
sarah.walker2_arm.com added a comment.Thu, Feb 26, 10:24 AM
In D55492#1270272, @br wrote:

looks like some part of vtnet still tries to pass physical addresses instead of bus dma addresses, so I get page faults

riscv_fq_intr
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940a iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940a iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940d iotval2: 0x0
riscv_iommu0: Fault: event 0xd received: Read page fault
riscv_iommu0: iotval 0x80e4940d iotval2: 0x0
riscv_iommu0: Fault: event 0xf received: Write/AMO page fault
riscv_iommu0: iotval 0x80e4940f iotval2: 0x0
riscv_iommu0: Fault: event 0xf received: Write/AMO page fault
riscv_iommu0: iotval 0x80e4940f iotval2: 0x0
vtnet0: cannot disable all-multicast mode

I'm getting a little confused now. Does vtnet work on RISC-V IOMMU without my changes?

br added a comment.Fri, Feb 27, 11:11 AM

I'm getting a little confused now. Does vtnet work on RISC-V IOMMU without my changes?

Yes, it did work in Bypass mode, and works with this patch as well in bypass.
With translation enabled it does not work before or after the patch, because some part of vtnet still passes physical addresses to HW (instead of busdma addresses).
Since you are converting vtnet to busdma, you could at least identify vtnet operations that are not yet converted.
This would benefit both RISC-V IOMMU and ARM System MMU as they work the same way.

also please keep lines 80 characters long per style(9) as some don't fit my terminal :)

sarah.walker2_arm.com updated this revision to Diff 172871.Fri, Feb 27, 3:59 PM

Fix set_iommu calls

Harbormaster completed remote builds in B71070: Diff 172871.Fri, Feb 27, 3:59 PM
sarah.walker2_arm.com added a comment.Fri, Feb 27, 4:00 PM
In D55492#1271245, @br wrote:

I'm getting a little confused now. Does vtnet work on RISC-V IOMMU without my changes?

Yes, it did work in Bypass mode, and works with this patch as well in bypass.
With translation enabled it does not work before or after the patch, because some part of vtnet still passes physical addresses to HW (instead of busdma addresses).
Since you are converting vtnet to busdma, you could at least identify vtnet operations that are not yet converted.
This would benefit both RISC-V IOMMU and ARM System MMU as they work the same way.

also please keep lines 80 characters long per style(9) as some don't fit my terminal :)

Could you try https://reviews.freebsd.org/D55564 (in addition to this change set) please?

br added a comment.Fri, Feb 27, 6:13 PM

Could you try https://reviews.freebsd.org/D55564 (in addition to this change set) please?

thanks, it works! Only small issue with large tag alignment

Revision Contents
Changeset List

PathSize
sys/
dev/
virtio/
network/
275 lines
10 lines

Diff 172871

View Options

sys/dev/virtio/network/if_vtnet.c

Loading...
View Options

sys/dev/virtio/network/if_vtnetvar.h

Loading...