Fix VF handling of VLANs
ClosedPublic
Actions

Authored by Petz_nisshoko.net on Jan 4 2016, 11:00 PM.

Details

Reviewers

scottl
gnn
sbruno
adrian

Group Reviewers

Intel Networking

Commits

rS293338: Fix VF handling of VLANs.

Summary

Bugzilla: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202983

This change aligns the FreeBSD driver behaviour closer to the Linux driver behaviour which is what I based the change off. In doing so it allows the driver to function on AWS instances that support Enhanced Networking.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

While I don't understand all the codebase of this Intel driver. The change seems quite straight forward.

When disabling the RX queue for updates in function ixv_initialize_receive_units. Don't remove VLAN support.
When running the function ixv_initialize_receive_units. It looks like we read the VFRXEDCTL twice which seems redundant given we have read it earlier in the function.
When enabling the RX queues, we should also flag them as VLAN capable with 'IXGBE_RXDCTL_VME'.

The linux driver also makes use of this FLAG during the RX queue initialisation process.
ixgbevf_main.c in 3.1.1 -> https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/3.1.1/

1936: 	/* allow any size packet since we can handle overflow */
1937: 	rxdctl &= ~IXGBE_RXDCTL_RLPML_EN;

But I'm not 100% sure on what it does. So am leaving this for a later change if it's needed. FreeBSd does seem to have this in the header files though but it never gets used from what I see. Anyway, I do wonder if this flag has anything to do with the automatic jumbo frame MTU detection and configuration as with this change alone the VF is setup and functions now, but does so with a default 1500 MTU instead of the 9001 which Linux systems do.

Test Plan

Have compile this and tested on an AWS M4 instance. Interface comes up now with DHCP successfully and I can connect.

root@ip-172-31-8-215:/usr/src # uname -a
FreeBSD ip-172-31-8-215 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r293137M: Mon Jan  4 11:24:40 UTC 2016     root@ip-172-31-8-215:/usr/obj/usr/src/sys/GENERIC  amd64

root@ip-172-31-8-215:/usr/src # dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r293137M: Mon Jan  4 11:24:40 UTC 2016
    root@ip-172-31-8-215:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.7.1 (tags/RELEASE_371/final 255217) 20151225
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): resolution 640x480
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (2400.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 8589934592 (8192 MB)
avail memory = 8270520320 (7887 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80edeed0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3008000-0xf3008fff at device 2.0 on pci0
vgapci0: Boot video device
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.4.6-k> mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: Ethernet address: 02:e2:5e:25:5e:dd
ixv0: netmap queues/slots: TX 1/1024, RX 1/1024
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 47 at device 31.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 20480MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
random: unblocking device.
ixv0: link state changed to UP

root@ip-172-31-8-215:/usr/src # ifconfig -a
ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
        ether 02:e2:5e:25:5e:dd
        inet 172.31.8.215 netmask 0xfffff000 broadcast 172.31.15.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

Petz_nisshoko.net updated this revision to Diff 11929.Jan 4 2016, 11:00 PM

Petz_nisshoko.net retitled this revision from to Fix VF handling of VLANs.

Petz_nisshoko.net updated this object.

Petz_nisshoko.net edited the test plan for this revision. (Show Details)

Petz_nisshoko.net added reviewers: scottl, adrian, gnn, sbruno.

Petz_nisshoko.net set the repository for this revision to rS FreeBSD src repository - subversion.

Petz_nisshoko.net added a subscriber: Petz_nisshoko.net.

Herald added a reviewer: Intel Networking. · View Herald TranscriptJan 4 2016, 11:00 PM

Herald added a subscriber: imp. · View Herald Transcript

Petz_nisshoko.net updated this object.Jan 4 2016, 11:02 PM

Petz_nisshoko.net edited the test plan for this revision. (Show Details)

Petz_nisshoko.net edited edge metadata.

Petz_nisshoko.net edited the test plan for this revision. (Show Details)

Petz_nisshoko.net updated this object.Jan 4 2016, 11:07 PM

Did some iperf3 tests to check how things perform. I believe without setting the MTU manually to 9001 first before testing, a MTU blackhole exists which ruins things. Not only does setting the MTU to 9001 manually fix the MTU blackhole reducing the retransmits(and UDP packet loss) but it also reduces CPU usage fairly significantly. It's a shame this isn't being set automatically.

# ifconfig ixv0 mtu 9001

I did three sets of tests. All tests run with m4.large instances in EC2 placement groups. Below is a summary

FreeBSD iperf client with xn(1500 MTU) -> Linux iperf server ixgbevf(9001 MTU)
- TCP: CPU ~35%, 439 Mbits/sec, 12567 Restransmits
- UDP: CPU ~25%, 400 Mbits/sec, Jitter 0.134 ms, Loss 491375/1223339 (40%)
FreeBSD iperf client with ixv(1500 MTU) -> Linux iperf server ixgbevf(9001 MTU)
- TCP: CPU ~35%, 439 Mbits/sec, 13248 Restransmits
- UDP: CPU ~19%, 400 Mbits/sec, Jitter 0.105 ms, Loss 573256/731477 (78%)
FreeBSD iperf client with ixv(9001 MTU) -> Linux iperf server ixgbevf(9001 MTU)
- TCP: CPU ~13%, 457 Mbits/sec, 2881 Restransmits
- UDP: CPU ~10%, 400 Mbits/sec, Jitter 0.049 ms, Loss 0/731908 (0%)

It is clear '3.' is ideal. It uses less than half the CPU. Drastrically reduces loss. Has around half as much jitter.

Also while not conclusive a simple ping of 20 packets shows response times are around half.

1. round-trip min/avg/max/stddev = 0.224/0.269/0.319/0.024 ms
2. Not tested
3. round-trip min/avg/max/stddev = 0.098/0.145/0.177/0.019 ms

ALL THE DETAILS BELOW

FreeBSD iperf client with xn(1500 MTU) -> Linux iperf server ixgbevf(9001 MTU)

iperf3 TCP top
--------------------------------------------------------------------------------
last pid:  1035;  load averages:  0.47,  0.22,  0.12                up 0+00:10:22  03:07:55
45 processes:  2 running, 43 sleeping
CPU:  2.2% user,  0.0% nice, 14.2% system, 18.5% interrupt, 65.2% idle
Mem: 35M Active, 25M Inact, 517M Wired, 158M Buf, 7369M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1035 root          1   4    0 10960K  2940K RUN     0   0:22  35.14% iperf3
  788 root          1  20    0 20096K  3972K CPU1    1   0:00   0.03% top
--------------------------------------------------------------------------------

iperf3 TCP
--------------------------------------------------------------------------------
root@ip-172-31-11-202:~ # iperf3 -c 172.31.13.127 -t 120 -i 30 -P 4
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.11.202 port 12446 connected to 172.31.13.127 port 5201
[  6] local 172.31.11.202 port 45965 connected to 172.31.13.127 port 5201
[  8] local 172.31.11.202 port 58564 connected to 172.31.13.127 port 5201
[ 10] local 172.31.11.202 port 29504 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-30.00  sec   246 MBytes  68.7 Mbits/sec  1095   32.0 MBytes
[  6]   0.00-30.00  sec   316 MBytes  88.4 Mbits/sec  837   62.0 MBytes
[  8]   0.00-30.00  sec   833 MBytes   233 Mbits/sec  331    122 MBytes
[ 10]   0.00-30.00  sec   272 MBytes  76.2 Mbits/sec  1003   24.0 MBytes
[SUM]   0.00-30.00  sec  1.63 GBytes   466 Mbits/sec  3266
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-60.00  sec   232 MBytes  64.9 Mbits/sec  1005   38.0 MBytes
[  6]  30.00-60.00  sec   295 MBytes  82.6 Mbits/sec  780   20.0 MBytes
[  8]  30.00-60.00  sec   768 MBytes   215 Mbits/sec  299    122 MBytes
[ 10]  30.00-60.00  sec   240 MBytes  67.1 Mbits/sec  998   46.0 MBytes
[SUM]  30.00-60.00  sec  1.50 GBytes   429 Mbits/sec  3082
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.00-90.00  sec   230 MBytes  64.2 Mbits/sec  1075   2.00 MBytes
[  6]  60.00-90.00  sec   287 MBytes  80.4 Mbits/sec  822   62.0 MBytes
[  8]  60.00-90.00  sec   767 MBytes   215 Mbits/sec  295   56.0 MBytes
[ 10]  60.00-90.00  sec   251 MBytes  70.1 Mbits/sec  961   50.0 MBytes
[SUM]  60.00-90.00  sec  1.50 GBytes   429 Mbits/sec  3153
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.00-120.00 sec   227 MBytes  63.4 Mbits/sec  1059   42.0 MBytes
[  6]  90.00-120.00 sec   294 MBytes  82.2 Mbits/sec  744   50.0 MBytes
[  8]  90.00-120.00 sec   766 MBytes   214 Mbits/sec  303   12.0 MBytes
[ 10]  90.00-120.00 sec   248 MBytes  69.4 Mbits/sec  960   14.0 MBytes
[SUM]  90.00-120.00 sec  1.50 GBytes   429 Mbits/sec  3066
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec   934 MBytes  65.3 Mbits/sec  4234             sender
[  4]   0.00-120.00 sec   934 MBytes  65.3 Mbits/sec                  receiver
[  6]   0.00-120.00 sec  1.17 GBytes  83.4 Mbits/sec  3183             sender
[  6]   0.00-120.00 sec  1.17 GBytes  83.4 Mbits/sec                  receiver
[  8]   0.00-120.00 sec  3.06 GBytes   219 Mbits/sec  1228             sender
[  8]   0.00-120.00 sec  3.06 GBytes   219 Mbits/sec                  receiver
[ 10]   0.00-120.00 sec  1012 MBytes  70.7 Mbits/sec  3922             sender
[ 10]   0.00-120.00 sec  1011 MBytes  70.7 Mbits/sec                  receiver
[SUM]   0.00-120.00 sec  6.13 GBytes   439 Mbits/sec  12567             sender
[SUM]   0.00-120.00 sec  6.13 GBytes   438 Mbits/sec                  receiver

iperf Done.
--------------------------------------------------------------------------------

iperf3 UDP top
--------------------------------------------------------------------------------
last pid:  1115;  load averages:  0.15,  0.18,  0.13                up 0+00:15:32  03:13:05
31 processes:  2 running, 29 sleeping
CPU:  0.4% user,  0.0% nice,  9.4% system,  2.6% interrupt, 87.6% idle
Mem: 29M Active, 25M Inact, 532M Wired, 174M Buf, 7360M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1115 root          1  31    0 10480K  2464K CPU0    0   0:07  19.08% iperf3
  788 root          1  20    0 20096K  3972K CPU1    1   0:00   0.04% top

--------------------------------------------------------------------------------

iperf3 UDP 4 x 100Mb/sec
--------------------------------------------------------------------------------
root@ip-172-31-11-202:~ # iperf3 -c 172.31.13.127 -u -t 120 -i 30 -P 4 -b 100M
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.11.202 port 46783 connected to 172.31.13.127 port 5201
[  6] local 172.31.11.202 port 47391 connected to 172.31.13.127 port 5201
[  8] local 172.31.11.202 port 32485 connected to 172.31.13.127 port 5201
[ 10] local 172.31.11.202 port 17918 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  150500
[  6]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  63496
[  8]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  48107
[ 10]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  49464
[SUM]   0.00-30.01  sec  1.39 GBytes   399 Mbits/sec  311567
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  147146
[  6]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  61711
[  8]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  48504
[ 10]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  48736
[SUM]  30.01-60.01  sec  1.40 GBytes   400 Mbits/sec  306097
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  142960
[  6]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  63225
[  8]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  47365
[ 10]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  48388
[SUM]  60.01-90.00  sec  1.40 GBytes   400 Mbits/sec  301938
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  145353
[  6]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  62185
[  8]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  47565
[ 10]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  48634
[SUM]  90.00-120.01 sec  1.40 GBytes   400 Mbits/sec  303737
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.122 ms  402971/585959 (69%)
[  4] Sent 585959 datagrams
[  6]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.134 ms  67625/250617 (27%)
[  6] Sent 250617 datagrams
[  8]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.144 ms  8549/191541 (4.5%)
[  8] Sent 191541 datagrams
[ 10]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.135 ms  12230/195222 (6.3%)
[ 10] Sent 195222 datagrams
[SUM]   0.00-120.01 sec  5.58 GBytes   400 Mbits/sec  0.134 ms  491375/1223339 (40%)                                                                                                                                                         

iperf Done.
--------------------------------------------------------------------------------

FreeBSD iperf client with ixv(1500 MTU) -> Linux iperf server ixgbevf(9001 MTU)

iperf3 TCP top MTU 1500
--------------------------------------------------------------------------------
last pid:  1075;  load averages:  0.63,  0.28,  0.20                                   up 0+00:36:37  03:04:50
45 processes:  1 running, 44 sleeping
CPU:  1.0% user,  0.0% nice, 17.1% system, 22.4% interrupt, 59.4% idle
Mem: 34M Active, 25M Inact, 518M Wired, 158M Buf, 7368M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1075 root          1  48    0 10960K  2936K select  0   0:09  35.17% iperf3
  829 root          1  20    0 20096K  3944K CPU1    1   0:00   0.04% top

--------------------------------------------------------------------------------

iperf3 TCP MTU 1500
--------------------------------------------------------------------------------
root@ip-172-31-4-176:~ # iperf3 -c 172.31.13.127 -t 120 -i 30 -P 4
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.4.176 port 26041 connected to 172.31.13.127 port 5201
[  6] local 172.31.4.176 port 39393 connected to 172.31.13.127 port 5201
[  8] local 172.31.4.176 port 13393 connected to 172.31.13.127 port 5201
[ 10] local 172.31.4.176 port 46507 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-30.00  sec   836 MBytes   234 Mbits/sec  367    120 MBytes
[  6]   0.00-30.00  sec   262 MBytes  73.2 Mbits/sec  1113   32.0 MBytes
[  8]   0.00-30.00  sec   290 MBytes  81.1 Mbits/sec  938   36.0 MBytes
[ 10]   0.00-30.00  sec   284 MBytes  79.5 Mbits/sec  1046   56.0 MBytes
[SUM]   0.00-30.00  sec  1.63 GBytes   468 Mbits/sec  3464
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-60.00  sec   768 MBytes   215 Mbits/sec  329   72.0 MBytes
[  6]  30.00-60.00  sec   245 MBytes  68.6 Mbits/sec  1049   42.0 MBytes
[  8]  30.00-60.00  sec   262 MBytes  73.2 Mbits/sec  913   22.0 MBytes
[ 10]  30.00-60.00  sec   261 MBytes  72.9 Mbits/sec  953   36.0 MBytes
[SUM]  30.00-60.00  sec  1.50 GBytes   429 Mbits/sec  3244
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.00-90.00  sec   768 MBytes   215 Mbits/sec  337    112 MBytes
[  6]  60.00-90.00  sec   247 MBytes  69.1 Mbits/sec  1000   48.0 MBytes
[  8]  60.00-90.00  sec   262 MBytes  73.1 Mbits/sec  940   46.0 MBytes
[ 10]  60.00-90.00  sec   259 MBytes  72.4 Mbits/sec  947   18.0 MBytes
[SUM]  60.00-90.00  sec  1.50 GBytes   429 Mbits/sec  3224
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.00-120.00 sec   768 MBytes   215 Mbits/sec  325   82.0 MBytes
[  6]  90.00-120.00 sec   254 MBytes  71.1 Mbits/sec  1051   16.0 MBytes
[  8]  90.00-120.00 sec   263 MBytes  73.6 Mbits/sec  952   60.0 MBytes
[ 10]  90.00-120.00 sec   250 MBytes  70.0 Mbits/sec  988   22.0 MBytes
[SUM]  90.00-120.00 sec  1.50 GBytes   429 Mbits/sec  3316
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  3.07 GBytes   219 Mbits/sec  1358             sender
[  4]   0.00-120.00 sec  3.07 GBytes   219 Mbits/sec                  receiver
[  6]   0.00-120.00 sec  1008 MBytes  70.5 Mbits/sec  4213             sender
[  6]   0.00-120.00 sec  1008 MBytes  70.5 Mbits/sec                  receiver
[  8]   0.00-120.00 sec  1.05 GBytes  75.3 Mbits/sec  3743             sender
[  8]   0.00-120.00 sec  1.05 GBytes  75.2 Mbits/sec                  receiver
[ 10]   0.00-120.00 sec  1.03 GBytes  73.7 Mbits/sec  3934             sender
[ 10]   0.00-120.00 sec  1.03 GBytes  73.7 Mbits/sec                  receiver
[SUM]   0.00-120.00 sec  6.13 GBytes   439 Mbits/sec  13248             sender
[SUM]   0.00-120.00 sec  6.13 GBytes   439 Mbits/sec                  receiver

iperf Done.
--------------------------------------------------------------------------------

iperf3 UDP top MTU 1500
--------------------------------------------------------------------------------
last pid:  1082;  load averages:  0.22,  0.33,  0.26                                   up 0+00:42:09  03:10:22
45 processes:  1 running, 44 sleeping
CPU:  0.5% user,  0.0% nice,  9.0% system,  4.7% interrupt, 85.8% idle
Mem: 33M Active, 26M Inact, 518M Wired, 158M Buf, 7368M Free
Swap:
 Command not understood
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1082 root          1  31    0 10480K  2460K select  1   0:03  18.88% iperf3
--------------------------------------------------------------------------------

iperf3 UDP 4 x 100Mb/sec MTU 1500
--------------------------------------------------------------------------------
root@ip-172-31-4-176:~ # iperf3 -c 172.31.13.127 -u -t 120 -i 30 -P 4 -b 100M
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.4.176 port 36607 connected to 172.31.13.127 port 5201
[  6] local 172.31.4.176 port 14446 connected to 172.31.13.127 port 5201
[  8] local 172.31.4.176 port 54081 connected to 172.31.13.127 port 5201
[ 10] local 172.31.4.176 port 20147 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-30.00  sec   357 MBytes  99.7 Mbits/sec  45661
[  6]   0.00-30.00  sec   357 MBytes  99.7 Mbits/sec  45661
[  8]   0.00-30.00  sec   357 MBytes  99.7 Mbits/sec  45661
[ 10]   0.00-30.00  sec   357 MBytes  99.7 Mbits/sec  45661
[SUM]   0.00-30.00  sec  1.39 GBytes   399 Mbits/sec  182644
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-60.01  sec   358 MBytes   100 Mbits/sec  45778
[  6]  30.00-60.01  sec   358 MBytes   100 Mbits/sec  45778
[  8]  30.00-60.01  sec   358 MBytes   100 Mbits/sec  45778
[ 10]  30.00-60.01  sec   358 MBytes   100 Mbits/sec  45778
[SUM]  30.00-60.01  sec  1.40 GBytes   400 Mbits/sec  183112
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  45781
[  6]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  45781
[  8]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  45781
[ 10]  60.01-90.00  sec   358 MBytes   100 Mbits/sec  45781
[SUM]  60.01-90.00  sec  1.40 GBytes   400 Mbits/sec  183124
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  45770
[  6]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  45770
[  8]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  45770
[ 10]  90.00-120.01 sec   358 MBytes   100 Mbits/sec  45770
[SUM]  90.00-120.01 sec  1.40 GBytes   400 Mbits/sec  183080
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.110 ms  142845/182870 (78%)
[  4] Sent 182870 datagrams
[  6]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.095 ms  143176/182869 (78%)
[  6] Sent 182869 datagrams
[  8]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.107 ms  143478/182869 (78%)
[  8] Sent 182869 datagrams
[ 10]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.107 ms  143757/182869 (79%)
[ 10] Sent 182869 datagrams
[SUM]   0.00-120.01 sec  5.58 GBytes   400 Mbits/sec  0.105 ms  573256/731477 (78%)

iperf Done.
--------------------------------------------------------------------------------

FreeBSD iperf client with ixv(9001 MTU) -> Linux iperf server ixgbevf(9001 MTU)

iperf3 TCP top MTU 9001
--------------------------------------------------------------------------------
last pid:  1170;  load averages:  0.23,  0.26,  0.24                                   up 0+00:50:44  03:18:57
31 processes:  1 running, 30 sleeping
CPU:  0.8% user,  0.0% nice,  6.7% system, 12.0% interrupt, 80.5% idle
Mem: 21M Active, 34M Inact, 541M Wired, 173M Buf, 7350M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1170 root          1  27    0 10960K  2940K select  0   0:05  12.94% iperf3
  829 root          1  20    0 20096K  3944K CPU1    1   0:00   0.03% top
--------------------------------------------------------------------------------

iperf3 TCP MTU 9001
--------------------------------------------------------------------------------
# iperf3 -c 172.31.13.127 -t 120 -i 30 -P 4
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.4.176 port 53719 connected to 172.31.13.127 port 5201
[  6] local 172.31.4.176 port 12060 connected to 172.31.13.127 port 5201
[  8] local 172.31.4.176 port 10660 connected to 172.31.13.127 port 5201
[ 10] local 172.31.4.176 port 46713 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-30.00  sec   869 MBytes   243 Mbits/sec  111   1.12 GBytes
[  6]   0.00-30.00  sec   310 MBytes  86.6 Mbits/sec  192   1.86 GBytes
[  8]   0.00-30.00  sec   285 MBytes  79.6 Mbits/sec  211    840 MBytes
[ 10]   0.00-30.00  sec   278 MBytes  77.8 Mbits/sec  229    611 MBytes
[SUM]   0.00-30.00  sec  1.70 GBytes   487 Mbits/sec  743
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-60.00  sec   799 MBytes   223 Mbits/sec  109    334 MBytes
[  6]  30.00-60.00  sec   257 MBytes  72.0 Mbits/sec  226    916 MBytes
[  8]  30.00-60.00  sec   275 MBytes  76.9 Mbits/sec  175    840 MBytes
[ 10]  30.00-60.00  sec   266 MBytes  74.5 Mbits/sec  202    916 MBytes
[SUM]  30.00-60.00  sec  1.56 GBytes   447 Mbits/sec  712
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.00-90.00  sec   799 MBytes   223 Mbits/sec  110   -450906448.00 Bytes
[  6]  60.00-90.00  sec   277 MBytes  77.3 Mbits/sec  173   1.12 GBytes
[  8]  60.00-90.00  sec   250 MBytes  69.9 Mbits/sec  238   1.12 GBytes
[ 10]  60.00-90.00  sec   272 MBytes  76.0 Mbits/sec  192   1.04 GBytes
[SUM]  60.00-90.00  sec  1.56 GBytes   447 Mbits/sec  713
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.00-120.00 sec   799 MBytes   223 Mbits/sec  110   -1492006261.00 Bytes
[  6]  90.00-120.00 sec   268 MBytes  75.0 Mbits/sec  190   1.72 GBytes
[  8]  90.00-120.00 sec   275 MBytes  77.0 Mbits/sec  175   1.04 GBytes
[ 10]  90.00-120.00 sec   255 MBytes  71.3 Mbits/sec  238    993 MBytes
[SUM]  90.00-120.00 sec  1.56 GBytes   447 Mbits/sec  713
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-120.00 sec  3.19 GBytes   228 Mbits/sec  440             sender
[  4]   0.00-120.00 sec  3.19 GBytes   228 Mbits/sec                  receiver
[  6]   0.00-120.00 sec  1.09 GBytes  77.7 Mbits/sec  781             sender
[  6]   0.00-120.00 sec  1.09 GBytes  77.7 Mbits/sec                  receiver
[  8]   0.00-120.00 sec  1.06 GBytes  75.8 Mbits/sec  799             sender
[  8]   0.00-120.00 sec  1.06 GBytes  75.8 Mbits/sec                  receiver
[ 10]   0.00-120.00 sec  1.05 GBytes  74.9 Mbits/sec  861             sender
[ 10]   0.00-120.00 sec  1.05 GBytes  74.9 Mbits/sec                  receiver
[SUM]   0.00-120.00 sec  6.38 GBytes   457 Mbits/sec  2881             sender
[SUM]   0.00-120.00 sec  6.38 GBytes   457 Mbits/sec                  receiver

iperf Done.
--------------------------------------------------------------------------------

iperf3 UDP top MTU 9001
--------------------------------------------------------------------------------
last pid:  1097;  load averages:  0.48,  0.32,  0.26                                   up 0+00:48:33  03:16:46
45 processes:  1 running, 44 sleeping
CPU:  0.0% user,  0.0% nice,  3.6% system,  3.6% interrupt, 92.7% idle
Mem: 25M Active, 33M Inact, 522M Wired, 158M Buf, 7365M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1097 root          1  23    0 10480K  2464K select  0   0:06   9.65% iperf3
  829 root          1  20    0 20096K  3944K CPU1    1   0:00   0.48% top

--------------------------------------------------------------------------------

iperf3 UDP 4 x 100Mb/sec MTU 9001
--------------------------------------------------------------------------------
root@ip-172-31-4-176:~ # iperf3 -c 172.31.13.127 -u -t 120 -i 30 -P 4 -b 100M
Connecting to host 172.31.13.127, port 5201
[  4] local 172.31.4.176 port 45155 connected to 172.31.13.127 port 5201
[  6] local 172.31.4.176 port 33141 connected to 172.31.13.127 port 5201
[  8] local 172.31.4.176 port 52846 connected to 172.31.13.127 port 5201
[ 10] local 172.31.4.176 port 31247 connected to 172.31.13.127 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  45645
[  6]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  45645
[  8]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  45645
[ 10]   0.00-30.01  sec   357 MBytes  99.7 Mbits/sec  45645
[SUM]   0.00-30.01  sec  1.39 GBytes   399 Mbits/sec  182580
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  45779
[  6]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  45779
[  8]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  45779
[ 10]  30.01-60.01  sec   358 MBytes   100 Mbits/sec  45779
[SUM]  30.01-60.01  sec  1.40 GBytes   400 Mbits/sec  183116
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  60.01-90.01  sec   358 MBytes   100 Mbits/sec  45775
[  6]  60.01-90.01  sec   358 MBytes   100 Mbits/sec  45775
[  8]  60.01-90.01  sec   358 MBytes   100 Mbits/sec  45775
[ 10]  60.01-90.01  sec   358 MBytes   100 Mbits/sec  45775
[SUM]  60.01-90.01  sec  1.40 GBytes   400 Mbits/sec  183100
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  90.01-120.01 sec   358 MBytes   100 Mbits/sec  45778
[  6]  90.01-120.01 sec   358 MBytes   100 Mbits/sec  45778
[  8]  90.01-120.01 sec   358 MBytes   100 Mbits/sec  45778
[ 10]  90.01-120.01 sec   358 MBytes   100 Mbits/sec  45778
[SUM]  90.01-120.01 sec  1.40 GBytes   400 Mbits/sec  183112
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.050 ms  0/182977 (0%)
[  4] Sent 182977 datagrams
[  6]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.048 ms  0/182977 (0%)
[  6] Sent 182977 datagrams
[  8]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.046 ms  0/182977 (0%)
[  8] Sent 182977 datagrams
[ 10]   0.00-120.01 sec  1.40 GBytes  99.9 Mbits/sec  0.052 ms  0/182977 (0%)
[ 10] Sent 182977 datagrams
[SUM]   0.00-120.01 sec  5.58 GBytes   400 Mbits/sec  0.049 ms  0/731908 (0%)

iperf Done.
--------------------------------------------------------------------------------

Having a look at my linux instance. I suspect it is not the Inytel driver setting the MTU to 9001 automatically. It looks to be dhclient.

On Linux dhclient seems to specifically have handling for the DHCP option for interface MTU.
http://tools.ietf.org/html/rfc2132#section-5.1

Looking at the linux /sbin/dhclient-script I see this

if [ "${reason}" = "BOUND" ] || [ "${reason}" = "REBOOT" ] ||
   [ ! "${old_ip_address}" = "${new_ip_address}" ] ||
   [ ! "${old_subnet_mask}" = "${new_subnet_mask}" ] ||
   [ ! "${old_network_number}" = "${new_network_number}" ] ||
   [ ! "${old_broadcast_address}" = "${new_broadcast_address}" ] ||
   [ ! "${old_routers}" = "${new_routers}" ] ||
   [ ! "${old_interface_mtu}" = "${new_interface_mtu}" ]; then
    ip -family inet addr add ${new_ip_address}/${new_prefix} broadcast ${new_broadcast_address} dev ${interface} >/dev/null 2>&1
    ip -family inet link set dev ${interface} up

    # The 576 MTU is only used for X.25 and dialup connections
    # where the admin wants low latency.  Such a low MTU can cause
    # problems with UDP traffic, among other things.  As such,
    # disallow MTUs from 576 and below by default, so that broken
    # MTUs are ignored, but higher stuff is allowed (1492, 1500, etc).
    if [ -n "${new_interface_mtu}" ] && [ ${new_interface_mtu} -gt 576 ]; then
        ip link set ${interface} mtu ${new_interface_mtu}

FreeBSD /sbin/dhclient-script does not seem to have handling for MTU so I suspect dhclient needs to be updated to handle this DHCP option.

Patrick or Pawel:

The current question is what dhclient and init scripts does with the MTU option. I don't see how its handled but I am ignorant in this space.

I'm guessing that we need to duplicate this behavior in /sbin/dhclient-script ?

sbruno@

I'd prefer that the dhcp issue not hold this up. I'll try to get another change created for that or log a bugzilla report should it prove beyond me.

I did do some digging on dhclient yesterday though. Judging by the header in the source file for dhclient.c, FreeBSD pulled this code from OpenBSD a while back.

1 	/*      $OpenBSD: dhclient.c,v 1.63 2005/02/06 17:10:13 krw Exp $       */

This is well before they added support for this DHCP option.

per http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/dhclient.c

Revision 1.357 / (download) - annotate - [select for diffs], Sat Feb 7 10:08:06 2015 UTC (10 months, 4 weeks ago) by krw
Branch: MAIN
Changes since 1.356: +13 -1 lines
Diff to previous 1.356 (colored)

Add support for interface-mtu (option 26).

Original request+diff from matthew@

ok dlg@

Note people seem to be doing different things with dhcp clients. For example, given the dhclient-script example I posted earlier from Amazon Linux, they seem to have this dhclient setting the mtu environment variables which it picks up and handles later in dhclient-script. This is different from OpenBSD which nuked dhclient-script over 3 years ago and now does everything from dhclient directly with ioctls

Revision 1.159 / (download) - annotate - [select for diffs], Tue Oct 30 18:39:44 2012 UTC (3 years, 2 months ago) by krw
Branch: MAIN
Changes since 1.158: +306 -427 lines
Diff to previous 1.158 (colored)

Nuke dhclient-script and dhclient.conf 'script' directive. Do all
interface and route configuration via ioctl's and routing sockets.

This will break configurations using local enhancements of
dhclient-script, which will now require alternate arrangements.

Committing early to allow time to identify and develop required
alternatives.

Several proddings by deraadt@.

NetBSD seems to be sync'ing fairly regularly with upstream sources and has two client implementations.

NetBSD dhclient: http://cvsweb.netbsd.org/bsdweb.cgi/src/external/bsd/dhcp/dist/client/#dirlist
NetBSD dhcpcd: http://cvsweb.netbsd.org/bsdweb.cgi/src/external/bsd/dhcpcd/dist/

Upstream Sources
dhclient: https://www.isc.org/downloads/dhcp/
dhcpcd: http://roy.marples.name/projects/dhcpcd/index

I guessing the best course of action would be for FreeBSD to also sync up more regularly with one or both of these sources as well in the base system. I can see that both of these packages are available via pkg. So perhaps I can test this out at some stage first.

root@ip-172-31-4-176:/usr/src/sys/dev/ixgbe # pkg search dhcp | grep -i client
dhcp6-20080615_3               KAME DHCP6 client, server, and relay
dhcpcd-6.9.4                   DHCP/IPv4LL/IPv6RS/DHCPv6 client   <-----
isc-dhcp41-client-4.1.e_2,2    The ISC Dynamic Host Configuration Protocol client
isc-dhcp42-client-4.2.8        The ISC Dynamic Host Configuration Protocol client
isc-dhcp43-client-4.3.3        The ISC Dynamic Host Configuration Protocol client <-----
p5-Net-DHCPClient-1.0_1        Perl implementation of a DHCP client API

Petz_nisshoko.net edited edge metadata.Jan 5 2016, 10:05 PM

Petz_nisshoko.net updated this object.

Looks good here.

This revision is now accepted and ready to land.Jan 6 2016, 3:00 PM

Petz_nisshoko.net edited edge metadata.Jan 7 2016, 5:54 AM

Petz_nisshoko.net added a subscriber: zi.

Incase your interested sbruno@.

I tested out the 'isc-dhcp43-client-4.3.3'. I managed to get it semi working.

By editing the /usr/local/etc/dhclient.conf and adding 'interface-mtu' to the 'request' section on line 6. It does get the MTU as part of the DHCP offer and dhclient-script processes that to set the interface. I added some logging lines to dhclient-script so I can see when it gets an MTU value.

root@ip-172-31-10-39:/usr/local/sbin # ./dhclient -q
dhclient: MTU not provided
dhclient: MTU not provided
dhclient: New Network Number: 172.31.0.0
dhclient: New Broadcast Address: 172.31.15.255
dhclient: New MTU: 9001
dhclient: New IP Address (ixv0): 172.31.10.39
dhclient: New Subnet Mask (ixv0): 255.255.240.0
dhclient: New Broadcast Address (ixv0): 172.31.15.255
dhclient: New Routers: 172.31.0.1

The part that's not working though is that the package setup instruction don't seem valid.

root@ip-172-31-10-39:~ # pkg install isc-dhcp43-client
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        isc-dhcp43-client: 4.3.3

The process will require 2 MiB more space.
603 KiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching isc-dhcp43-client-4.3.3.txz: 100%  603 KiB 308.9kB/s    00:02
Checking integrity... done (0 conflicting)
[1/1] Installing isc-dhcp43-client-4.3.3...
[1/1] Extracting isc-dhcp43-client-4.3.3: 100%
Message from isc-dhcp43-client-4.3.3:
****  To setup dhclient, you may need to edit /etc/rc.conf to replace the
      base system dhclient as follows:

            dhclient_program="/usr/local/sbin/dhclient"
            dhclient_flags="-q"

      See dhclient(8), using the following command, for details about other
      possible options:

            MANPATH=/usr/local/man man 8 dhclient

Following those setup instructions ends up with an EC2 instance that doesn't work. I seems like the DHCP client runs. But I am assuming from reading other posts about a similar issue it runs a second time after devd is notified of the link-up event. I'm not sure if this second run of dhclient triggered by devd is killing the programmed interface and routes or if that LOR stack trace has something to do with it. Console output below.

I'm not sure what to do with this, do I need to disable the dhclient rc.d script? Disable dhclient in devd? Remove SYNCDHCP from rc.conf? Perhaps I'm wrong altogether and running it a second time isn't all that bad and the problem is with the LOR stack trace? (found potential worakround below, don;t know why though)

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r293137M: Mon Jan  4 11:24:40 UTC 2016
    root@ip-172-31-8-215:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.7.1 (tags/RELEASE_371/final 255217) 20151225
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): resolution 640x480
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (2400.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 8589934592 (8192 MB)
avail memory = 8270520320 (7887 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80edeed0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3008000-0xf3008fff at device 2.0 on pci0
vgapci0: Boot video device
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.4.6-k> mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: Ethernet address: 02:16:8f:d9:11:0f
ixv0: netmap queues/slots: TX 1/1024, RX 1/1024
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 47 at device 31.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 20480MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Setting hostuuid: ec290971-3211-e1dd-81fd-c32df78e8d74.
Setting hostid: 0x2cf051fc.
No suitable dump device was found.
Starting file system checks:
/dev/gpt/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/gpt/rootfs: clean, 2237660 free (75884 frags, 270222 blocks, 1.5% fragmentation)
Mounting local file systems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/llvm36/lib
random: unblocking device.
32-bit compatibility ldconfig path: /usr/lib32
Feeding entropy:.
Starting dhclient.
dhclient: MTU not provided
dhclient: MTU not provided
ixv0: link state changed to UP
dhclient: New Network Number: 172.31.0.0
dhclient: New Broadcast Address: 172.31.15.255
dhclient: New MTU: 9001
dhclient: New Hostname: ip-172-31-10-39
dhclient: New IP Address (ixv0): 172.31.10.39
dhclient: New Subnet Mask (ixv0): 255.255.240.0
dhclient: New Broadcast Address (ixv0): 172.31.15.255
dhclient: New Routers: 172.31.0.1
lock order reversal:
 1st 0xfffffe01f0e23930 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3476
 2nd 0xfffff800070ee800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:281
stack backtrace:
#0 0xffffffff80a799f0 at witness_debugger+0x70
#1 0xffffffff80a798f1 at witness_checkorder+0xe71
#2 0xffffffff80a26ea2 at _sx_xlock+0x72
#3 0xffffffff80cc0f5d at ufsdirhash_add+0x3d
#4 0xffffffff80cc3e0f at ufs_direnter+0x62f
#5 0xffffffff80ccd173 at ufs_makeinode+0x5f3
#6 0xffffffff80cc8d1d at ufs_create+0x2d
#7 0xffffffff80fb5be1 at VOP_CREATE_APV+0xf1
#8 0xffffffff80ae1ff8 at vn_open_cred+0x2f8
#9 0xffffffff80adb37c at kern_openat+0x25c
#10 0xffffffff80e6b13b at amd64_syscall+0x2db
#11 0xffffffff80e4a5ab at Xfast_syscall+0xfb
lock order reversal:
 1st 0xfffff80007043b78 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2473
 2nd 0xfffffe01f0e23930 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:263
 3rd 0xfffff8000766d9a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2473
stack backtrace:
#0 0xffffffff80a799f0 at witness_debugger+0x70
#1 0xffffffff80a798f1 at witness_checkorder+0xe71
#2 0xffffffff809fb0db at __lockmgr_args+0xd3b
#3 0xffffffff80cbbfe6 at ffs_lock+0xa6
#4 0xffffffff80fb8180 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae288a at _vn_lock+0x9a
#6 0xffffffff80ad2cf3 at vget+0x63
#7 0xffffffff80ac54fc at vfs_hash_get+0xcc
#8 0xffffffff80cb77b0 at ffs_vgetf+0x40
#9 0xffffffff80caf141 at softdep_sync_buf+0xad1
#10 0xffffffff80cbcbc6 at ffs_syncvnode+0x256
#11 0xffffffff80c93b7d at ffs_truncate+0x6cd
#12 0xffffffff80cc3f9b at ufs_direnter+0x7bb
#13 0xffffffff80ccd173 at ufs_makeinode+0x5f3
#14 0xffffffff80cc8d1d at ufs_create+0x2d
#15 0xffffffff80fb5be1 at VOP_CREATE_APV+0xf1
#16 0xffffffff80ae1ff8 at vn_open_cred+0x2f8
#17 0xffffffff80adb37c at kern_openat+0x25c
Starting Network: lo0 ixv0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001
	options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
	ether 02:16:8f:d9:11:0f
	inet 172.31.10.39 netmask 0xfffff000 broadcast 172.31.15.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
Starting devd.
Starting dhclient.
dhclient: MTU not provided
/etc/rc.d/dhclient: WARNING: failed to start dhclient
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Starting syslogd.
fetch: http://169.254.169.254/latest/meta-data/block-device-mapping/: Operation timed out
No ephemeral disks are available, so no swap space is being created.
Starting casperd.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Performing sanity check on sshd configuration.
Starting sshd.
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 dc:e8:14:d2:f0:a9:41:7a:f4:29:4f:d1:bd:8a:81:e3  root@ip-172-31-8-215 (DSA)
ec2: 256 2a:1b:60:43:a9:6d:66:29:bc:e0:26:f4:1b:5d:6e:9a  root@ip-172-31-8-215 (ECDSA)
ec2: 256 38:43:3f:02:6e:cd:e7:45:04:7e:d9:b2:52:59:b7:ba  root@ip-172-31-8-215 (ED25519)
ec2: 2048 be:33:ea:b3:f3:0b:1a:91:2c:f1:35:3e:40:61:44:90  root@ip-172-31-8-215 (RSA1)
ec2: 2048 c3:6a:ff:93:ce:67:71:76:e8:e8:a7:93:7f:49:4e:84  root@ip-172-31-8-215 (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
Starting sendmail_submit.
Starting sendmail_msp_queue.
Configuring vt: blanktime.
Starting cron.
Starting background file system checks in 60 seconds.
Thu Jan  7 06:05:21 UTC 2016
FreeBSD/amd64 (ip-172-31-10-39) (ttyu0)
login:

Looks like the answer is remove SYNCDHCP from rc.conf. The below is exactly what I had before in rc.conf but I just changed SYNCDHCP to DHCP for 'ifconfig_DEFAULT'

# cat /etc/rc.conf
dhclient_program="/usr/local/sbin/dhclient"
dhclient_flags="-q"

ec2_configinit_enable=YES
ec2_fetchkey_enable=YES
ec2_ephemeralswap_enable=YES
ec2_loghostkey_enable=YES

firstboot_freebsd_update_enable=YES
firstboot_pkgs_enable=YES
growfs_enable="YES"

ifconfig_DEFAULT="DHCP"
sshd_enable="YES"
firstboot_pkgs_list="awscli"

Result is that it works. But is this just luck that devd started before dhclient(ie. race condition)?

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r293137M: Mon Jan  4 11:24:40 UTC 2016
    root@ip-172-31-8-215:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.7.1 (tags/RELEASE_371/final 255217) 20151225
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): resolution 640x480
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (2394.50-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 8589934592 (8192 MB)
avail memory = 8270520320 (7887 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80edeed0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3008000-0xf3008fff at device 2.0 on pci0
vgapci0: Boot video device
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.4.6-k> mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: Ethernet address: 02:16:8f:d9:11:0f
ixv0: netmap queues/slots: TX 1/1024, RX 1/1024
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 47 at device 31.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 20480MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Setting hostuuid: ec290971-3211-e1dd-81fd-c32df78e8d74.
Setting hostid: 0x2cf051fc.
No suitable dump device was found.
Starting file system checks:
/dev/gpt/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/gpt/rootfs: clean, 2237632 free (75880 frags, 270219 blocks, 1.5% fragmentation)
Mounting local file systems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/llvm36/lib
random: unblocking device.
32-bit compatibility ldconfig path: /usr/lib32
Feeding entropy:.
ixv0: link state changed to UP
Starting Network: lo0 ixv0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
	ether 02:16:8f:d9:11:0f
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
Starting devd.
Starting dhclient.
dhclient: MTU not provided
dhclient: MTU not provided
dhclient: New Network Number: 172.31.0.0
dhclient: New Broadcast Address: 172.31.15.255
dhclient: New MTU: 9001
dhclient: New Hostname: ip-172-31-10-39
dhclient: New IP Address (ixv0): 172.31.10.39
dhclient: New Subnet Mask (ixv0): 255.255.240.0
dhclient: New Broadcast Address (ixv0): 172.31.15.255
dhclient: New Routers: 172.31.0.1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Starting syslogd.
No ephemeral disks are available, so no swap space is being created.
Starting casperd.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Performing sanity check on sshd configuration.
Starting sshd.
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 dc:e8:14:d2:f0:a9:41:7a:f4:29:4f:d1:bd:8a:81:e3  root@ip-172-31-8-215 (DSA)
ec2: 256 2a:1b:60:43:a9:6d:66:29:bc:e0:26:f4:1b:5d:6e:9a  root@ip-172-31-8-215 (ECDSA)
ec2: 256 38:43:3f:02:6e:cd:e7:45:04:7e:d9:b2:52:59:b7:ba  root@ip-172-31-8-215 (ED25519)
ec2: 2048 be:33:ea:b3:f3:0b:1a:91:2c:f1:35:3e:40:61:44:90  root@ip-172-31-8-215 (RSA1)
ec2: 2048 c3:6a:ff:93:ce:67:71:76:e8:e8:a7:93:7f:49:4e:84  root@ip-172-31-8-215 (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
Starting sendmail_submit.
Starting sendmail_msp_queue.
Configuring vt: blanktime.
Starting cron.
Starting background file system checks in 60 seconds.
Thu Jan  7 05:55:52 UTC 2016
FreeBSD/amd64 (ip-172-31-10-39) (ttyu0)
login: Jan  7 05:56:08 ip-172-31-10-39 su: ec2-user to root on /dev/pts/0

Going to commit this today. Is this a potential MFC to stable/10 candidate?

rstone added a subscriber: rstone.Jan 7 2016, 6:30 PM

Closed by commit rS293338: Fix VF handling of VLANs. (authored by sbruno). · Explain WhyJan 7 2016, 6:35 PM

This revision was automatically updated to reflect the committed changes.

Petz_nisshoko.net added a subscriber: erj.Jan 7 2016, 10:18 PM

In D4788#102332, @sbruno wrote:

Is this a potential MFC to stable/10 candidate?

Not sure if you directed this at me Sean. But it would be awesome if we could get this functioning(MFCed) in FreeBSD 10 at least. This change is not all that's required though. Before this change Intel(or Eric Joyner erj@) dropped a big diff here which fixed some broader issues in the driver. As such, you would want to MFC that diff also for this one to have an impact. I see your comments for the commit of this diff you specified MFC as 2 weeks, but the below diff & commit doesn't mention any MFC so I'm not sure if that will happen?

https://reviews.freebsd.org/D4186
https://reviews.freebsd.org/rS292674