Sleep rather than spin in e1000 when doing long running config operations

[kmacy]

Summary

Title says it all. Depends on D14983.

[bcr]

When I read the title of the review, I thought it was some kind of medical advice for sysadmins or something. ;-)
Sorry, this does not help the review, just wanted to let you know.

[erj]

In D14984#315219, @bcr wrote:

When I read the title of the review, I thought it was some kind of medical advice for sysadmins or something. ;-)
Sorry, this does not help the review, just wanted to let you know.

The real advice is to not use e1000 devices if you're a sysadmin. :p

(j/k)

[shurd]

Minor nitpick

sys/dev/e1000/e1000_osdep.h

71

This macro doesn't seem to be used.

[sbruno]

Basic compile, boot tests are good to go. Perf TBD

[dumbbell]

Since this commit, I hit a panic during boot.

Before the panic, I get the following warning, displayed a dozen times:

Sleeping on "e1000_delay" with the following non-sleepable locks held:
exclusive rm if_lagg rmlock (if_lagg rmlock) r = 0 (0xfffff8001124fc08) locked @ /home/dumbbell/Projects/freebsd/src/GIT/sys/net/if_lagg.c:1433
stack backtrace:
#0 0xffffffff80bde2a3 at witness_debugger+0x73
#1 0xffffffff80bdf681 at witness_warn+0x461
#2 0xffffffff80b80c48 at _sleep+0x58
#3 0xffffffff80b814c4 at pause_sbt+0x144
#4 0xffffffff805f2ab4 at e1000_write_phy_reg_mdic+0xf4
#5 0xffffffff805f5fa4 at e1000_access_phy_wakeup_reg_bm+0x74
#6 0xffffffff805f674f at __e1000_write_phy_reg_hv+0x9f
#7 0xffffffff805e4c8e at e1000_update_mc_addr_list_pch2lan+0xae
#8 0xffffffff805c614f at em_if_multi_set+0x1bf
#9 0xffffffff80c96a7e at iflib_if_ioctl+0xfe
#10 0xffffffff82d8aa15 at lagg_ioctl+0x115
#11 0xffffffff80c79384 at if_addmulti+0x264
#12 0xffffffff80db8466 at in6_joingroup_locked+0x1d6
#13 0xffffffff80db8264 at in6_joingroup+0x44
#14 0xffffffff80db1495 at in6_update_ifa+0x1665
#15 0xffffffff80db5ad3 at in6_ifattach+0x513
#16 0xffffffff80c7663a at if_up+0x6a
#17 0xffffffff80c787a7 at ifioctl+0x1e27

Then the panic:

Sleeping thread (tid 100427, pid 312) owns a non-sleepable lock
KDB: stack backtrace of thread 100427:
sched_switch() at sched_switch+0x942/frame 0xfffffe008c757ee0
mi_switch() at mi_switch+0x18c/frame 0xfffffe008c757f10
sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe008c757f50
sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe008c757f90
_sleep() at _sleep+0x34d/frame 0xfffffe008c758030
pause_sbt() at pause_sbt+0x144/frame 0xfffffe008c758080
e1000_write_phy_reg_mdic() at e1000_write_phy_reg_mdic+0xf4/frame 0xfffffe008c7580c0
e1000_access_phy_wakeup_reg_bm() at e1000_access_phy_wakeup_reg_bm+0x179/frame 0xfffffe008c758100
__e1000_write_phy_reg_hv() at __e1000_write_phy_reg_hv+0x9f/frame 0xfffffe008c758150
e1000_update_mc_addr_list_pch2lan() at e1000_update_mc_addr_list_pch2lan+0xae/frame 0xfffffe008c758190
em_if_multi_set() at em_if_multi_set+0x1bf/frame 0xfffffe008c7581e0
iflib_if_ioctl() at iflib_if_ioctl+0xfe/frame 0xfffffe008c758250
lagg_ioctl() at lagg_ioctl+0x115/frame 0xfffffe008c758300
if_addmulti() at if_addmulti+0x264/frame 0xfffffe008c758390
in6_joingroup_locked() at in6_joingroup_locked+0x1d6/frame 0xfffffe008c758430
in6_joingroup() at in6_joingroup+0x44/frame 0xfffffe008c758460
in6_update_ifa() at in6_update_ifa+0x1665/frame 0xfffffe008c758610
in6_ifattach() at in6_ifattach+0x513/frame 0xfffffe008c758750
if_up() at if_up+0x6a/frame 0xfffffe008c758780
ifioctl() at ifioctl+0x1e27/frame 0xfffffe008c758850
kern_ioctl() at kern_ioctl+0x2b9/frame 0xfffffe008c7588b0
sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe008c758980
amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c758ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe008c758ab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800468b2a, rsp = 0x7fffffffe428, rbp = 0x7fffffffe480 ---
panic: sleeping thread
cpuid = 4
time = 1526134087
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000040f120
vpanic() at vpanic+0x1a3/frame 0xfffffe000040f180
panic() at panic+0x43/frame 0xfffffe000040f1e0
propagate_priority() at propagate_priority+0x335/frame 0xfffffe000040f220
turnstile_wait() at turnstile_wait+0x38d/frame 0xfffffe000040f270
__mtx_lock_sleep() at __mtx_lock_sleep+0x1e1/frame 0xfffffe000040f2f0
__mtx_lock_flags() at __mtx_lock_flags+0xf9/frame 0xfffffe000040f340
_rm_rlock_hard() at _rm_rlock_hard+0x1d3/frame 0xfffffe000040f370
_rm_rlock_debug() at _rm_rlock_debug+0x14c/frame 0xfffffe000040f3c0
lagg_transmit() at lagg_transmit+0x38/frame 0xfffffe000040f430
ether_output_frame() at ether_output_frame+0xaa/frame 0xfffffe000040f460
ether_output() at ether_output+0x68f/frame 0xfffffe000040f500
ip6_output() at ip6_output+0x1f7c/frame 0xfffffe000040f780
mld_dispatch_packet() at mld_dispatch_packet+0x302/frame 0xfffffe000040f800
mld_fasttimo() at mld_fasttimo+0xb02/frame 0xfffffe000040f8d0
pffasttimo() at pffasttimo+0x54/frame 0xfffffe000040f900
softclock_call_cc() at softclock_call_cc+0x150/frame 0xfffffe000040f9b0
softclock() at softclock+0x7c/frame 0xfffffe000040f9e0
intr_event_execute_handlers() at intr_event_execute_handlers+0x99/frame 0xfffffe000040fa20
ithread_loop() at ithread_loop+0xb7/frame 0xfffffe000040fa70
fork_exit() at fork_exit+0x84/frame 0xfffffe000040fab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000040fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 5s
Dumping 784 out of 16028 MB:..3%..11%..21%..31%..41%..52%..62%..72%..82%..92%

__curthread () at ./machine/pcpu.h:231
231     ./machine/pcpu.h: No such file or directory.
(kgdb) #0  __curthread () at ./machine/pcpu.h:231
#1  doadump (textdump=1)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_shutdown.c:366
#2  0xffffffff80b76772 in kern_reboot (howto=260)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_shutdown.c:446
#3  0xffffffff80b76d53 in vpanic (fmt=<optimized out>, ap=0xfffffe000040f1c0)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_shutdown.c:863
#4  0xffffffff80b76da3 in panic (fmt=<unavailable>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_shutdown.c:790
#5  0xffffffff80bd3f45 in propagate_priority (td=0xfffff80007f89000)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/subr_turnstile.c:228
#6  0xffffffff80bd49fd in turnstile_wait (ts=0xfffff80003640b40, 
    owner=0xfffff80007f89000, queue=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/subr_turnstile.c:783
#7  0xffffffff80b55aa1 in __mtx_lock_sleep (c=0xfffff8001124fc60, 
    v=<optimized out>, opts=<optimized out>, file=<optimized out>, 
    line=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_mutex.c:639
#8  0xffffffff80b55829 in __mtx_lock_flags (c=0xfffff8001124fc60, 
    opts=<optimized out>, file=<optimized out>, line=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_mutex.c:255
#9  0xffffffff80b708c3 in _rm_rlock_hard (rm=0xfffff8001124fc08, 
    tracker=0xfffffe000040f3d8, trylock=0)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_rmlock.c:410
#10 0xffffffff80b7120c in _rm_rlock_debug (rm=0xfffff8001124fc08, 
    tracker=0xfffffe000040f3d8, trylock=0, 
    file=0xffffffff82d935d0 "/home/dumbbell/Projects/freebsd/src/GIT/sys/net/if_lagg.c", line=1643)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_rmlock.c:659
#11 0xffffffff82d8a708 in lagg_transmit (ifp=0xfffff8000b0b3000, 
    m=0xfffff800132b7800)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/net/if_lagg.c:1643
#12 0xffffffff80c7df1a in ether_output_frame (ifp=0xfffff8000b0b3000, 
    m=<unavailable>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/net/if_ethersubr.c:481
#13 0xffffffff80c7ddbf in ether_output (ifp=0xfffff8000b0b3000, 
    m=<unavailable>, dst=0xfffffe000040f668, ro=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/net/if_ethersubr.c:435
#14 0xffffffff80dc708c in ip6_output (m0=<optimized out>, 
    opt=0xffffffff81f9dfc8 <mld_po>, ro=0x0, flags=1, 
    im6o=0xfffffe000040f798, ifpp=0xfffffe000040f7d0, inp=0x0)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/netinet6/ip6_output.c:956
#15 0xffffffff80dcdeb2 in mld_dispatch_packet (m=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/netinet6/mld6.c:3132
#16 0xffffffff80dcbaf2 in mld_dispatch_queue (limit=0, mq=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/netinet6/mld6.c:429
#17 mld_fasttimo_vnet ()
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/netinet6/mld6.c:1438
#18 mld_fasttimo ()
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/netinet6/mld6.c:1316
#19 0xffffffff80c046a4 in pffasttimo (arg=<unavailable>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/uipc_domain.c:521
#20 0xffffffff80b8edc0 in softclock_call_cc (
    c=0xffffffff81f97d50 <pffast_callout>, cc=0xffffffff81ffdd80 <cc_cpu>, 
    direct=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_timeout.c:731
#21 0xffffffff80b8f18c in softclock (arg=0xffffffff81ffdd80 <cc_cpu>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_timeout.c:869
#22 0xffffffff80b394b9 in intr_event_execute_handlers (p=<optimized out>, 
    ie=0xfffff8000365c000)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_intr.c:1338
#23 0xffffffff80b39ba7 in ithread_execute_handlers (ie=<optimized out>, 
    p=<optimized out>)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_intr.c:1351
#24 ithread_loop (arg=0xfffff80003662100)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_intr.c:1432
#25 0xffffffff80b368f4 in fork_exit (
    callout=0xffffffff80b39af0 <ithread_loop>, arg=0xfffff80003662100, 
    frame=0xfffffe000040fac0)
    at /home/dumbbell/Projects/freebsd/src/GIT/sys/kern/kern_fork.c:1039
#26 <signal handler called>

The current workaround is to disable the use of lagg(4).