Do not generate illegal mbuf chains during IP fragment reassembly.

[np]

Summary

Do not generate illegal mbuf chains during IP fragment reassembly. Only
the first mbuf of the reassembled datagram should have a pkthdr.

This was discovered with cxgbe(4) + IPSEC + ping with payload more than
interface MTU. cxgbe can generate !M_WRITEABLE mbufs and this results
in m_unshare being called on the reassembled datagram, and it complains:

panic: m_unshare: m0 0xfffff80020f82600, m 0xfffff8005d054100 has M_PKTHDR

[ae]

I think the description that you gave in IRC was better :)
I.e. a !WRITABLE mbuf is reflected by ICMP code.

[np]

In D14009#293983, @ae wrote:

I think the description that you gave in IRC was better :)
I.e. a !WRITABLE mbuf is reflected by ICMP code.

Even M_WRITEABLE mbufs are reflected but m_unshare is called only for !M_WRITEABLE
ones. Here's the stack showing the reflection and the problem I'm trying to fix:

panic: m_unshare: m0 0xfffff80020f82600, m 0xfffff8005d054100 has M_PKTHDR
cpuid = 15
time = 1495578455
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe044e9bb890
kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe044e9bb960
vpanic() at vpanic+0x269/frame 0xfffffe044e9bba30
kassert_panic() at kassert_panic+0xc7/frame 0xfffffe044e9bbac0
m_unshare() at m_unshare+0x578/frame 0xfffffe044e9bbbc0
esp_output() at esp_output+0x44c/frame 0xfffffe044e9bbe40
ipsec4_perform_request() at ipsec4_perform_request+0x5df/frame 0xfffffe044e9bbff0
ipsec4_process_packet() at ipsec4_process_packet+0x28/frame 0xfffffe044e9bc020
ipsec4_common_output() at ipsec4_common_output+0x101/frame 0xfffffe044e9bc070
ipsec4_output() at ipsec4_output+0x69/frame 0xfffffe044e9bc0b0
ip_output() at ip_output+0x1322/frame 0xfffffe044e9bc380
icmp_send() at icmp_send+0x114/frame 0xfffffe044e9bc3e0
icmp_reflect() at icmp_reflect+0x7ff/frame 0xfffffe044e9bc4f0
icmp_input() at icmp_input+0x93a/frame 0xfffffe044e9bc6f0
ip_input() at ip_input+0xd44/frame 0xfffffe044e9bc840
netisr_process_workstream_proto() at netisr_process_workstream_proto+0x211/frame 0xfffffe044e9bc8f0
swi_net() at swi_net+0x107/frame 0xfffffe044e9bc930
intr_event_execute_handlers() at intr_event_execute_handlers+0x252/frame 0xfffffe044e9bc980
ithread_execute_handlers() at ithread_execute_handlers+0x47/frame 0xfffffe044e9bc9b0
ithread_loop() at ithread_loop+0x106/frame 0xfffffe044e9bca30
fork_exit() at fork_exit+0x145/frame 0xfffffe044e9bcab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe044e9bcab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

[np]

ugh. I still need reviewers for this. I accidentally accepted it as myself and on behalf of network.

[ae]

I think the description that you gave in IRC was better :)
I.e. a !WRITABLE mbuf is reflected by ICMP code.

Even M_WRITEABLE mbufs are reflected but m_unshare is called only for !M_WRITEABLE
ones. Here's the stack showing the reflection and the problem I'm trying to fix:

Yes, I meant that it would be good to mention that mbuf was made on inbound path, but the assert happens on outbound. And it is not quite obvious how it happens :)
But this trace also good enough, much better that previous in mailing list.