We see constant tx hangs when changing driver settings:

On client:

ping <SUT>

on SUT:

#kldload if_ixl
#ifconfig ixl0 <IPv4 address>

Client is now seeing ICMP replies:
64 bytes from 190.2.8.77: icmp_seq=33 ttl=64 time=0.162 ms
64 bytes from 190.2.8.77: icmp_seq=34 ttl=64 time=0.060 ms
64 bytes from 190.2.8.77: icmp_seq=35 ttl=64 time=0.057 ms
64 bytes from 190.2.8.77: icmp_seq=36 ttl=64 time=0.057 ms
64 bytes from 190.2.8.77: icmp_seq=37 ttl=64 time=0.057 ms
64 bytes from 190.2.8.77: icmp_seq=38 ttl=64 time=0.052 ms
64 bytes from 190.2.8.77: icmp_seq=39 ttl=64 time=0.067 ms
64 bytes from 190.2.8.77: icmp_seq=40 ttl=64 time=0.054 ms
64 bytes from 190.2.8.77: icmp_seq=41 ttl=64 time=0.051 ms
64 bytes from 190.2.8.77: icmp_seq=42 ttl=64 time=0.056 ms
64 bytes from 190.2.8.77: icmp_seq=43 ttl=64 time=0.058 ms

On SUT:

ifconfig ixl0 down up

ICMP replies on client stop.

On SUT:

#dmesg
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0
ixl0: TX(10) desc avail = 1, pidx = 0

This can also be reproduced by:

#kldload if_ixl
#ifconfig ixl0 <IP address>
#ifconfig ixl0 <IP address> mtu 9000

Somehow my last comment got lost. With real Netflix traffic, the sparse detection logic is not sufficient. When we have kernel TLS, which inserts 13 and 16 byte mbufs around TLS records, we hit MDDs regularly. I think you need to walk the packet stream, not the DMA descriptors, so that you keep the same segmentation used by the hardware. I re-wrote the sparse detection to do this, and our MDDs went away.

This is the sparse detect I'm using. Note how it walks the header, and then the TSO by packet on the wire.

static int
ixl_tso_detect_sparse(bus_dma_segment_t *segs, int nsegs, if_pkt_info_t pi)
{
        int     count, curseg, i, hlen, segsz, seglen, tsolen;

        if (nsegs <= IXL_MAX_TX_SEGS-2)
                return (0);
        segsz = pi->ipi_tso_segsz;
        curseg = count = 0;

        hlen = pi->ipi_ehdrlen + pi->ipi_ip_hlen + pi->ipi_tcp_hlen;
        tsolen = pi->ipi_len - hlen;

        i = 0;
        curseg = segs[0].ds_len;
        while (hlen > 0) {
                count++;
                if (count > IXL_MAX_TX_SEGS - 2)
                        return (1);
                if (curseg == 0) {
                        i++;
                        if (__predict_false(i == nsegs))
                                return (1);

                        curseg = segs[i].ds_len;
                }
                seglen = min(curseg, hlen);
                curseg -= seglen;
                hlen -= seglen;
                printf("H:seglen = %d, count=%d\n", seglen, count);
        }
        while (tsolen > 0) {
                segsz = pi->ipi_tso_segsz;
                while (segsz > 0 && tsolen > 0) {
                        count++;
                        if (count > IXL_MAX_TX_SEGS - 2)
                                return (1);
                        if (curseg == 0) {
                                i++;
                                if (__predict_false(i == nsegs))
                                        return (1);
                                curseg = segs[i].ds_len;
                        }
                        seglen = min(curseg, segsz);
                        segsz -= seglen;
                        curseg -= seglen;
                        tsolen -= seglen;
                        printf("D:seglen = %d, count=%d\n", seglen, count);
                }
                count = 0;
        }

        return (0);
}

This also needs to be rebased again , probably due to: https://reviews.freebsd.org/D15343:

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)

cmdline: git rebase master
stdout: 'First, rewinding head to replay your work on top of it...

Applying: Update ixl(4) to use iflib.
Using index info to reconstruct a base tree...
M sys/net/iflib.c
Falling back to patching base and 3-way merge...
Auto-merging sys/net/iflib.c
CONFLICT (content): Merge conflict in sys/net/iflib.c
Patch failed at 0001 Update ixl(4) to use iflib.
Use 'git am --show-current-patch' to see the failed patch

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
'

stderr: '.git/rebase-apply/patch:56: trailing whitespace.
Copyright (c) 2013-2015, Intel Corporation

.git/rebase-apply/patch:58: trailing whitespace.

.git/rebase-apply/patch:59: trailing whitespace.

Redistribution and use in source and binary forms, with or without

.git/rebase-apply/patch:61: trailing whitespace.

.git/rebase-apply/patch:62: trailing whitespace.

1. Redistributions of source code must retain the above copyright notice,

warning: squelched 81 whitespace errors
warning: 86 lines add whitespace errors.
error: Failed to merge in the changes.'

This applied and could be rebased cleanly before D15343 was committed.

You might want to go for some middle ground here. 1ms is a bit much (even for us).

In D15577#334003, @gallatin wrote:

You might want to go for some middle ground here. 1ms is a bit much (even for us).

I remember another company using that value for their default ITR values, but that might have been for the VF driver.

I think we need to lower it. 500 significantly impacts 40G performance w/ 9K MTU, especially rx.

Any last-minute objections? This is the version that's going in!