Scenario:
virtual machine has 2 interfaces (vlan11, vlan12) on top of vtnet0.
We're building lagg on top of there 2 interfaces (see checklagg.sh for exact details).
Kernel is hacked to
* allow vlans on top of lagg (one-liner)
* help dtrace probes by explicitly uninlining several functions
* dummy arp_iflladdr() lladdr_event callback added to ease dtrace handling
Files:
http://static.ipfw.ru/patches/d4004_checklagg.sh
http://static.ipfw.ru/patches/d4004_arp.d
http://static.ipfw.ru/patches/d4004_kernel.diff
checklagg.sh is running along with arp.d script in other console. Their output is merged and some comments are provided.
============================================================================================
============================================================================================
============================================================================================
# TESTCASE 1, HEAD (r289939)
[14:10:41] Working on:{P85}
vlan11 52:54:11:11:11:11# TESTCASE 2, PATCHED VERSION
vlan12 52:54:12:12:12:12
[14:10:41] Creating lagg lagg0 on top of vlan11
[14:10:41] Testing inherited mac.. OK{P85}
CPU ID FUNCTION:NAMEfiles:
## Queing notification request for lagg mac changeF270951
3 60254 lagg_lladdr:entry Setting 'lagg0' mac to 52:54:11:11:11:11F270953
3 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 52:54:11:11:11:11F270955
0 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
2 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## eventhandler should have been called for 'lagg0'
2 17375 arp_iflladdr:entry ARP handler called for 'vlan11'
[14:10:46] Creating vlan devices (vlan900, vlan901) on lagg0.. DONE
[14:10:46] Checking if interfaces has been created with proper macs..
vlan900 OK
vlan901 OK
[14:10:51] Adding (slave) port vlan12 to lagg0.. DONE
[14:10:51] Checking if slave port mac has changed.. OK
## Enqueueing slave port lladdr change
0 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:11:11:11:11
2 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## MAC set but NO notification
2 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:11:11:11:11
[14:10:56] Removing primary port from lagg (vlan12 is new primary).. DONE
[14:10:56] Checking if all interfaces has updated their macs..
vlan11 OK
vlan12 OK
lagg0 OK
vlan900 FAIL
vlan901 FAIL
## Enqueue lladdr notification for removed interface
0 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
## Choose new mac and change lagg lladdr/enqueue change
0 60254 lagg_lladdr:entry Setting 'lagg0' mac to 52:54:12:12:12:12
0 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 52:54:12:12:12:12
## Enqueue TWO DIFFERENT (llq_primary) mac addr change for remaining port (new primary)
0 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
0 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
2 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Handling queue backward
## Send notification for vlan12 (this line is for llq_primary message)
2 17375 arp_iflladdr:entry ARP handler called for 'vlan12'
## And this one is for 'no one is primary' cycle
2 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
## it was primary at the moment of insertion, so notification only (OK)
2 17375 arp_iflladdr:entry ARP handler called for 'vlan11'
[14:11:01] Adding (slave) port vlan11 back.. DONE
[14:11:01] Checking if slave port mac has changed.. OK
## Enqueue lladdr change for newly-attached child interface
1 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:12:12:12:12
2 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Set mac BUT NO notification
2 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:12:12:12:12
[14:11:06] Trying to change mac manually to 52:54:00:00:00:00.. DONE
[14:11:06] Checking if all interfaces has updated their macs..
vlan11 OK
vlan12 FAIL
lagg0 OK
vlan900 OK
vlan901 OK
## Start from setlladdr -> lagg_stop() -> lagg_init()
2 33396 if_setlladdr:entry Setting 'lagg0' mac to 52:54:00:00:00:00
## lagg_init() enqueues change for each port
2 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:00:00:00:00
2 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:00:00:00:00
## eventhnadler notification from if_setlladdr() on cpu2
2 33396 if_setlladdr:entry Setting 'vlan901' mac to 52:54:00:00:00:00
2 33396 if_setlladdr:entry Setting 'vlan900' mac to 52:54:00:00:00:00
2 17375 arp_iflladdr:entry ARP handler called for 'lagg0'
## taskqueue handler on cpu 0
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## NOT setting mac for vlan12 because it is "primary" but send notify
0 17375 arp_iflladdr:entry ARP handler called for 'vlan12'
## Set mac BUT don't send notification..
0 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:00:00:00:00
[14:11:11] Cleaning up.. DONE
[14:11:11] Checking original macs..
vlan11 OK
vlan12 OK
2 60218 lagg_clone_destroy:entry time to shutdown for 'lagg0'
## Enqueing delete for vlan11 (slave port)
2 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
## Takqueue triggered, waiting for lock?
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Enqueing delete for vlan12 (the only remaining master)
2 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
2 60254 lagg_lladdr:entry Setting 'lagg0' mac to 00:00:00:00:00:00
2 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 00:00:00:00:00:00
## Send notify for vlan12 (mac is already correct)
0 17375 arp_iflladdr:entry ARP handler called for 'vlan12'
## Set mac BUT don't send notification..
0 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
## tasqueue_drain() for empty queue?
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
============================================================================================
============================================================================================
============================================================================================
# TESTCASE 2, PATCHED VERSION
[22:02:02] Working on:
vlan11 52:54:11:11:11:11
vlan12 52:54:12:12:12:12
[22:02:02] Creating lagg lagg0 on top of vlan11
[22:02:02] Testing inherited mac.. OK
CPU ID FUNCTION:NAME
## Queing notification request for lagg mac change
0 60254 lagg_lladdr:entry Setting 'lagg0' mac to 52:54:11:11:11:11
0 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 52:54:11:11:11:11
1 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## No forced update (hence no down/up), just eventhandler
1 17375 arp_iflladdr:entry ARP handler called for 'lagg0' [mac 52:54:11:11:11:11]
## Ignored set request (same lladdr for physical port)
0 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
[22:02:07] Creating vlan devices (vlan900, vlan901) on lagg0.. DONE
[22:02:07] Checking if interfaces has been created with proper macs..
vlan900 OK
vlan901 OK
[22:02:12] Adding (slave) port vlan12 to lagg0.. DONE
[22:02:12] Checking if slave port mac has changed.. OK
## Enqueueing slave port lladdr change
0 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:11:11:11:11
1 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Set mac + send notification
1 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:11:11:11:11
1 17375 arp_iflladdr:entry ARP handler called for 'vlan12' [mac 52:54:11:11:11:11]
[22:02:17] Removing primary port from lagg (vlan12 is new primary).. DONE
[22:02:17] Checking if all interfaces has updated their macs..
vlan11 OK
vlan12 OK
lagg0 OK
vlan900 OK
vlan901 OK
## Enqueue lladdr change for removed interface (ignored since lladdr was not changed)
0 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
## Choose new mac and change lagg lladdr/enqueue change
0 60254 lagg_lladdr:entry Setting 'lagg0' mac to 52:54:12:12:12:12
0 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 52:54:12:12:12:12
0 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
1 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Handling queue backward
## Set mac + send notification for vlan12
1 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
1 17375 arp_iflladdr:entry ARP handler called for 'vlan12' [mac 52:54:12:12:12:12]
## Executing event handlers for lagg0 (not setting lladdr )
1 33396 if_setlladdr:entry Setting 'vlan901' mac to 52:54:12:12:12:12
1 33396 if_setlladdr:entry Setting 'vlan900' mac to 52:54:12:12:12:12
1 17375 arp_iflladdr:entry ARP handler called for 'lagg0' [mac 52:54:12:12:12:12]
[22:02:22] Adding (slave) port vlan11 back.. DONE
[22:02:22] Checking if slave port mac has changed.. OK
## Enqueue lladdr change for newly-attached child interface
1 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:12:12:12:12
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Set mac + send notification for vlan11
0 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:12:12:12:12
0 17375 arp_iflladdr:entry ARP handler called for 'vlan11' [mac 52:54:12:12:12:12]
[22:02:27] Trying to change mac manually to 52:54:00:00:00:00.. DONE
[22:02:27] Checking if all interfaces has updated their macs..
vlan11 OK
vlan12 OK
lagg0 OK
vlan900 OK
vlan901 OK
## Start from setlladdr -> lagg_stop() -> lagg_init()
2 33396 if_setlladdr:entry Setting 'lagg0' mac to 52:54:00:00:00:00
## lagg_init() enqueues change for each port
2 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:00:00:00:00
2 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:00:00:00:00
## taskqueue handler on cpu0
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## eventhnadler notification from if_setlladdr() on cpu2
2 33396 if_setlladdr:entry Setting 'vlan901' mac to 52:54:00:00:00:00
## Set mac + send notification for vlan12
0 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:00:00:00:00
0 17375 arp_iflladdr:entry ARP handler called for 'vlan12' [mac 52:54:00:00:00:00]
## Set mac + send notification for vlan11
0 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:00:00:00:00
0 17375 arp_iflladdr:entry ARP handler called for 'vlan11' [mac 52:54:00:00:00:00]
## eventhnadler notification from if_setlladdr() on cpu0
2 33396 if_setlladdr:entry Setting 'vlan900' mac to 52:54:00:00:00:00
2 17375 arp_iflladdr:entry ARP handler called for 'lagg0' [mac 52:54:00:00:00:00]
[22:02:32] Cleaning up.. DONE
[22:02:32] Checking original macs..
vlan11 OK
vlan12 OK
1 60218 lagg_clone_destroy:entry time to shutdown for 'lagg0'
## Enqueing delete for vlan11 (slave port)
1 60270 lagg_port_lladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
## Takqueue triggered, waiting for lock?
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'
## Enqueing delete for vlan12 (the only remaining master)
1 60270 lagg_port_lladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
## lagg_port_destroy(): no ports, set empty lladdr
1 60254 lagg_lladdr:entry Setting 'lagg0' mac to 00:00:00:00:00:00
1 60270 lagg_port_lladdr:entry Setting 'lagg0' mac to 00:00:00:00:00:00
## iterating taskq queue backwards
## Send notification for lagg0
0 17375 arp_iflladdr:entry ARP handler called for 'lagg0' [mac 00:00:00:00:00:00]
## Set mac + send notification for vlan12
0 33396 if_setlladdr:entry Setting 'vlan12' mac to 52:54:12:12:12:12
0 17375 arp_iflladdr:entry ARP handler called for 'vlan12' [mac 52:54:12:12:12:12]
## Set mac + send notification for vlan11
0 33396 if_setlladdr:entry Setting 'vlan11' mac to 52:54:11:11:11:11
0 17375 arp_iflladdr:entry ARP handler called for 'vlan11' [mac 52:54:11:11:11:11]
## tasqueue_drain() for empty queue?
0 60274 lagg_port_setlladdr:entry Taskqueue handler called for 'lagg0'