Fix netmap + vlan panics

[afedorov]

Summary

It is very useful to be able to create virtual networks, such as: VM1 - VALE switch - if_vlan - ix0 -- physical network -- ix0 - if_vlan - VALE switch - VM2.

This patch fixes two panic.

First panic:

root@current:~ # ifconfig vlan create vlan 1024 vlandev vtnet0 up
vlan0
root@current:~ # pkt-gen -i vlan0 -f tx
611.182750 main [2889] interface is vlan0
611.183004 main [3011] using default burst size: 512
611.183021 main [3019] running on 1 cpus (have 4)
611.185667 extract_ip_range [471] range is 10.0.611.186530 [1129] generic_netmap_attach     Emulated adapter for vlan0 created (prev was 0)
0.1:1234 to 10.0.0.1:1234
611.185897 extract_ip_range [471] range is 10.1.0.1:1234 to 10.1.0.1:1234
611.185980 nm_open [858] overriding ARG1 0
611.186215 nm_open [862] overriding ARG2 0
611.186227 nm_open [866] overriding ARG3 0
611.186237 nm_open [870] overriding RING_CFG
611.186248 nm_open [879] overriding ifname vlan0 ringid 0x0 flags 0x8001
611.385901 [ 320] generic_netmap_register   Emulated adapter for vlan0 activated
611.387781 main [3117] mapped 334980KB at 0x800e00000
Sending on netmap:vlan0: 1 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
611.389515 main [3224] Sending 512 packets every  0.000000000 s
611.389636 start_threads [2549] Wait 2 secs for phy reset
613.435285 start_threads [2551] Ready...
613.435737 sender_body [1580] start, fd 3 main_fd 3
613.436545 sender_body [1638] frags 1 frag_

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80cd5f2e
stack pointer           = 0x28:0xfffffe001971d520
frame pointer           = 0x28:0xfffffe001971d550
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
			processor eflags        = interrupt enabled, resume, IOPL = 0
			current process         = 732 (pkt-gen)
			trap number             = 12
			panic: page fault
			cpuid = 1
			time = 1561989613
			KDB: stack backtrace:
			db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe001971d1e0
			vpanic() at vpanic+0x19d/frame 0xfffffe001971d230
			panic() at panic+0x43/frame 0xfffffe001971d290
			trap_fatal() at trap_fatal+0x39c/frame 0xfffffe001971d2f0
			trap_pfault() at trap_pfault+0x62/frame 0xfffffe001971d340
			trap() at trap+0x2b4/frame 0xfffffe001971d450
			calltrap() at calltrap+0x8/frame 0xfffffe001971d450
			--- trap 0xc, rip = 0xffffffff80cd5f2e, rsp = 0xfffffe001971d520, rbp = 0xfffffe001971d550 ---
			ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe001971d550
			vlan_transmit() at vlan_transmit+0xd4/frame 0xfffffe001971d5c0
			nm_os_generic_xmit_frame() at nm_os_generic_xmit_frame+0x48/frame 0xfffffe001971d5d0
			generic_netmap_txsync() at generic_netmap_txsync+0x25d/frame 0xfffffe001971d6a0
			netmap_poll() at netmap_poll+0x324/frame 0xfffffe001971d770
			freebsd_netmap_poll() at freebsd_netmap_poll+0x32/frame 0xfffffe001971d7a0
			devfs_poll_f() at devfs_poll_f+0x71/frame 0xfffffe001971d7f0
			kern_poll() at kern_poll+0x3d9/frame 0xfffffe001971d970
			sys_poll() at sys_poll+0x50/frame 0xfffffe001971d990
			amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe001971dab0
			fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe001971dab0
			--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8004943aa, rsp = 0x7fffdfffde48, rbp = 0x7fffdfffde80 ---
			KDB: enter: panic
			[ thread pid 732 tid 100137 ]
			Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
			db>

Netmap generic directly calls the if_transmit method of the underlying driver (https://svnweb.freebsd.org/base/head/sys/dev/netmap/netmap_freebsd.c?revision=348022&view=markup#l455). Panic occurs at https://svnweb.freebsd.org/base/head/sys/net/if_ethersubr.c?revision=348254&view=markup#l1371 due to the fact that the current VNET is not set. The panic disappears if the VIMAGE option is disabled in the config.

To correct this panic, set the current VNET to the driver VNET.

The second panic:

root@current:~ # pkt-gen -i vlan0 -f tx
372.565904 main [2889] interface is vlan0
372.566203 main [3011] using default burst size: 512
372.566513 main [3019] running on 1 cpus (have 4)
372.568995 extract_ip_range [471] range is 10.0.0.1:1234 to 10.0.0.1:1234
372.569272 extract_ip_range [471] ran372.571191 [1129] generic_netmap_attach     Emulated adapter for vlan0 created (prev was 0)
ge is 10.1.0.1:1234 to 10.1.0.1:1234
372.569632 nm_open [858] overriding ARG1 0
372.569930 nm_open [862] overriding ARG2 0
372.570231 nm_open [866] overriding ARG3 0
372.570534 nm_open [870] overriding RING_CFG
372.570863 nm_open [879] overriding ifname vlan0 ringid 0x0 flags 0x8001
372.785305 [ 320] generic_netmap_register   Emulated adapter for vlan0 activated
372.787260 main [3117] mapped 334980KB at 0x800e00000
Sending on netmap:vlan0: 1 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
372.788704 main [3224] Sending 512 packets every  0.000000000 s
372.789062 start_threads [2549] Wait 2 secs for phy reset
374.865353 start_threads [2551] Ready...
374.866365 sender_body [1580] start, fd 3 main_fd 3
374.866666 sender_body [1638] frags 1 frag_size 60
panic: vtnet_txq_encap: no mbuf packet header!
cpuid = 0
time = 1561994374
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0019732380
vpanic() at vpanic+0x19d/frame 0xfffffe00197323d0
panic() at panic+0x43/frame 0xfffffe0019732430
vtnet_txq_encap() at vtnet_txq_encap+0x3ba/frame 0xfffffe00197324a0
vtnet_txq_mq_start_locked() at vtnet_txq_mq_start_locked+0x126/frame 0xfffffe0019732500
vtnet_txq_mq_start() at vtnet_txq_mq_start+0x70/frame 0xfffffe0019732540
vlan_transmit() at vlan_transmit+0xe6/frame 0xfffffe00197325b0
nm_os_generic_xmit_frame() at nm_os_generic_xmit_frame+0x78/frame 0xfffffe00197325d0
generic_netmap_txsync() at generic_netmap_txsync+0x25d/frame 0xfffffe00197326a0
netmap_poll() at netmap_poll+0x324/frame 0xfffffe0019732770
freebsd_netmap_poll() at freebsd_netmap_poll+0x32/frame 0xfffffe00197327a0
devfs_poll_f() at devfs_poll_f+0x71/frame 0xfffffe00197327f0
kern_poll() at kern_poll+0x3d9/frame 0xfffffe0019732970
sys_poll() at sys_poll+0x50/frame 0xfffffe0019732990
amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0019732ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0019732ab0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8004943aa, rsp = 0x7fffdfffde48, rbp = 0x7fffdfffde80 ---
KDB: enter: panic
[ thread pid 779 tid 100113 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
db>

Netmap uses a preallocated array of mbuf's to send packets through the underlying driver. These mbuf's are allocated and initialized during the creation of the netmap adapter, in particular, the M_PKTHDR flag is set for them. After completing sending, netmap tries to reuse these mbuf's, but some network drivers (if_vlan, if_vxlan) may reset the M_PKTHDR flag. This leads to a panic on M_ASSERTPKTHDR (m) at line: https://svnweb.freebsd.org/base/head/sys/dev/virtio/network/if_vtnet.c?revision=348599&view=markup#l2201

To fix the second panic, restore M_PKTHDR flag before call if_transmit method.

Test Plan

pkt-gen test through vlan beetween real machines over physical network to check first panic:

1. Machine
# ifconfig vlan create vlan10 vlandev ix0 up
vlan0
# pkt-gen -i vlan0 -f tx
2. Machine 
# ifconfig vlan create vlan10 vlandev ix0 up
vlan0
# pkt-gen -i vlan0 -f rx

pkt-gen test beetween VM's on same machine to check the second panic:

VM1:
# ifconfig vlan create vlan10 vlandev vtnet0 up
vlan0
# pkt-gen -i vlan0 -f tx
VM2:
# ifconfig vlan create vlan10 vlandev vtnet0 up
vlan0
# pkt-gen -i vlan0 -f rx

P/S: We use netmap + vlan with these fixes for more than two months in preproduction environment. Without any problems.

[vmaffione]

Looks good, thanks.
The netmap unit tests and integration tests still pass with these changes.

As a side, using the netmap emulated adapter (aka generic) is rather slow, because of the translation layer between the netmap buffers and the mbufs. The right way to let netmap interact with VLANs would be to implement the VLAN strip/push operations in your netmap application, and always open physical ports (e.g. ix0). I know this is hard to do here, because the VALE switch is not easy to extend in kernel-space. In theory VALE could be ported to userspace, so that you can easily add custom logic for VLAN handling.