Index: head/share/man/man4/sfxge.4 =================================================================== --- head/share/man/man4/sfxge.4 (revision 294076) +++ head/share/man/man4/sfxge.4 (revision 294077) @@ -1,177 +1,179 @@ .\" Copyright (c) 2011-2015 Solarflare Communications Inc. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions are met: .\" .\" 1. Redistributions of source code must retain the above copyright notice, .\" this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright notice, .\" this list of conditions and the following disclaimer in the documentation .\" and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" .\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, .\" THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR .\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR .\" CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, .\" EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, .\" PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; .\" OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, .\" WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, .\" EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .\" .\" The views and conclusions contained in the software and documentation are .\" those of the authors and should not be interpreted as representing official .\" policies, either expressed or implied, of the FreeBSD Project. .\" .\" $FreeBSD$ .\" .Dd February 22, 2015 .Dt SFXGE 4 .Os .Sh NAME .Nm sfxge .Nd "Solarflare 10Gb Ethernet adapter driver" .Sh SYNOPSIS To compile this driver into the kernel, place the following lines in your kernel configuration file: .Bd -ragged -offset indent .Cd "device sfxge" .Ed .Pp To load the driver as a module at boot time, place the following line in .Xr loader.conf 5 : .Bd -literal -offset indent sfxge_load="YES" .Ed .Sh DESCRIPTION The .Nm driver provides support for 10Gb Ethernet adapters based on Solarflare SFC9000 family controllers. The driver supports jumbo frames, transmit/receive checksum offload, TCP Segmentation Offload (TSO), Large Receive Offload (LRO), VLAN checksum offload, VLAN TSO, and Receive Side Scaling (RSS) using MSI-X interrupts. .Pp The driver allocates 1 receive queue, transmit queue, event queue and IRQ per CPU up to a maximum of 64. IRQ affinities should be spread out using .Xr cpuset 1 . Interrupt moderation may be controlled through the sysctl .Va dev.sfxge.%d.int_mod (units are microseconds). .Pp For more information on configuring this device, see .Xr ifconfig 8 . .Pp A large number of MAC, PHY and data path statistics are available under the sysctl .Va dev.sfxge.%d.stats . The adapter's VPD fields including its serial number are available under the sysctl .Va dev.sfxge.%d.vpd . .Sh HARDWARE The .Nm driver supports all 10Gb Ethernet adapters based on Solarflare SFC9000 family controllers. .Sh LOADER TUNABLES Tunables can be set at the .Xr loader 8 prompt before booting the kernel or stored in .Xr loader.conf 5 . Actual values can be obtained using .Xr sysctl 8 . .Bl -tag -width indent .It Va hw.sfxge.rx_ring The maximum number of descriptors in a receive queue ring. Supported values are: 512, 1024, 2048 and 4096. .It Va hw.sfxge.tx_ring The maximum number of descriptors in a transmit queue ring. Supported values are: 512, 1024, 2048 and 4096. .It Va hw.sfxge.tx_dpl_get_max The maximum length of the deferred packet .Dq get-list for queued transmit packets (TCP and non-TCP), used only if the transmit queue lock can be acquired. If a packet is dropped, the .Va tx_get_overflow counter is incremented and the local sender receives ENOBUFS. The value must be greater than 0. .It Va hw.sfxge.tx_dpl_get_non_tcp_max The maximum number of non-TCP packets in the deferred packet .Dq get-list , used only if the transmit queue lock can be acquired. If a packet is dropped, the .Va tx_get_non_tcp_overflow counter is incremented and the local sender receives ENOBUFS. The value must be greater than 0. .It Va hw.sfxge.tx_dpl_put_max The maximum length of the deferred packet .Dq put-list for queued transmit packets, used if the transmit queue lock cannot be acquired. If a packet is dropped, the .Va tx_put_overflow counter is incremented and the local sender receives ENOBUFS. The value must be greater than or equal to 0. .It Va hw.sfxge.tso_fw_assisted -Enable/disable usage of FW-assisted TSO if supported by NIC firmware. -Enabled by default. +Bitmask to enable/disable usage of FW-assisted TSO version if supported +by NIC firmware. +FATSOv1 (bit 0) and FATSOv2 (bit 1) are supported. +All enabled by default. .It Va hw.sfxge.N.max_rss_channels The maximum number of allocated RSS channels for the Nth adapter. If set to 0 or unset, the number of channels is determined by the number of CPU cores. .It Va hw.sfxge.lro.table_size Size of the LRO hash table. Must be a power of 2. A larger table means we can accelerate a larger number of streams. .It Va hw.sfxge.lro.chain_max The maximum length of a hash chain. If chains get too long then the lookup time increases and may exceed the benefit of LRO. .It Va hw.sfxge.lro.idle_ticks The maximum time (in ticks) that a connection can be idle before it's LRO state is discarded. .It Va hw.sfxge.lro.slow_start_packets Number of packets with payload that must arrive in-order before a connection is eligible for LRO. The idea is we should avoid coalescing segments when the sender is in slow-start because reducing the ACK rate can damage performance. .It Va hw.sfxge.lro.loss_packets Number of packets with payload that must arrive in-order following loss before a connection is eligible for LRO. The idea is we should avoid coalescing segments when the sender is recovering from loss, because reducing the ACK rate can damage performance. .It Va hw.sfxge.mcdi_logging Enable logging of MCDI protocol messages (only available if enabled at compile-time). .It Va hw.sfxge.N.mcdi_logging Enable or disable logging of MCDI protocol messages on a per-port basis. The default for each port will be the value of .Va hw.sfxge.mcdi_logging. The logging may also be enabled or disabled after the driver is loaded using the sysctl .Va dev.sfxge.%d.mcdi_logging. .El .Sh SUPPORT For general information and support, go to the Solarflare support website at: .Pa https://support.solarflare.com . .Sh SEE ALSO .Xr cpuset 1 , .Xr arp 4 , .Xr netintro 4 , .Xr ng_ether 4 , .Xr vlan 4 , .Xr ifconfig 8 .Sh AUTHORS The .Nm driver was written by .An Philip Paeps and .An Solarflare Communications, Inc. Index: head/sys/dev/sfxge/sfxge.h =================================================================== --- head/sys/dev/sfxge/sfxge.h (revision 294076) +++ head/sys/dev/sfxge/sfxge.h (revision 294077) @@ -1,435 +1,438 @@ /*- * Copyright (c) 2010-2015 Solarflare Communications Inc. * All rights reserved. * * This software was developed in part by Philip Paeps under contract for * Solarflare Communications, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The views and conclusions contained in the software and documentation are * those of the authors and should not be interpreted as representing official * policies, either expressed or implied, of the FreeBSD Project. * * $FreeBSD$ */ #ifndef _SFXGE_H #define _SFXGE_H #include #include #include #include #include #include #include #include #include #include #include #include "sfxge_ioc.h" /* * Debugging */ #if 0 #define DBGPRINT(dev, fmt, args...) \ device_printf(dev, "%s: " fmt "\n", __func__, ## args) #else #define DBGPRINT(dev, fmt, args...) #endif /* * Backward-compatibility */ #ifndef CACHE_LINE_SIZE /* This should be right on most machines the driver will be used on, and * we needn't care too much about wasting a few KB per interface. */ #define CACHE_LINE_SIZE 128 #endif #ifndef IFCAP_LINKSTATE #define IFCAP_LINKSTATE 0 #endif #ifndef IFCAP_VLAN_HWTSO #define IFCAP_VLAN_HWTSO 0 #endif #ifndef IFM_10G_T #define IFM_10G_T IFM_UNKNOWN #endif #ifndef IFM_10G_KX4 #define IFM_10G_KX4 IFM_10G_CX4 #endif #ifndef IFM_40G_CR4 #define IFM_40G_CR4 IFM_UNKNOWN #endif #if (__FreeBSD_version >= 800501 && __FreeBSD_version < 900000) || \ __FreeBSD_version >= 900003 #define SFXGE_HAVE_DESCRIBE_INTR #endif #ifdef IFM_ETH_RXPAUSE #define SFXGE_HAVE_PAUSE_MEDIAOPTS #endif #ifndef CTLTYPE_U64 #define CTLTYPE_U64 CTLTYPE_QUAD #endif #include "sfxge_rx.h" #include "sfxge_tx.h" #define ROUNDUP_POW_OF_TWO(_n) (1ULL << flsl((_n) - 1)) #define SFXGE_IP_ALIGN 2 #define SFXGE_ETHERTYPE_LOOPBACK 0x9000 /* Xerox loopback */ enum sfxge_evq_state { SFXGE_EVQ_UNINITIALIZED = 0, SFXGE_EVQ_INITIALIZED, SFXGE_EVQ_STARTING, SFXGE_EVQ_STARTED }; #define SFXGE_EV_BATCH 16384 struct sfxge_evq { /* Structure members below are sorted by usage order */ struct sfxge_softc *sc; struct mtx lock; unsigned int index; enum sfxge_evq_state init_state; efsys_mem_t mem; efx_evq_t *common; unsigned int read_ptr; boolean_t exception; unsigned int rx_done; unsigned int tx_done; /* Linked list of TX queues with completions to process */ struct sfxge_txq *txq; struct sfxge_txq **txqs; /* Structure members not used on event processing path */ unsigned int buf_base_id; unsigned int entries; char lock_name[SFXGE_LOCK_NAME_MAX]; } __aligned(CACHE_LINE_SIZE); #define SFXGE_NDESCS 1024 #define SFXGE_MODERATION 30 enum sfxge_intr_state { SFXGE_INTR_UNINITIALIZED = 0, SFXGE_INTR_INITIALIZED, SFXGE_INTR_TESTING, SFXGE_INTR_STARTED }; struct sfxge_intr_hdl { int eih_rid; void *eih_tag; struct resource *eih_res; }; struct sfxge_intr { enum sfxge_intr_state state; struct resource *msix_res; struct sfxge_intr_hdl *table; int n_alloc; int type; efsys_mem_t status; uint32_t zero_count; }; enum sfxge_mcdi_state { SFXGE_MCDI_UNINITIALIZED = 0, SFXGE_MCDI_INITIALIZED, SFXGE_MCDI_BUSY, SFXGE_MCDI_COMPLETED }; struct sfxge_mcdi { struct mtx lock; efsys_mem_t mem; enum sfxge_mcdi_state state; efx_mcdi_transport_t transport; /* Only used in debugging output */ char lock_name[SFXGE_LOCK_NAME_MAX]; }; struct sfxge_hw_stats { clock_t update_time; efsys_mem_t dma_buf; void *decode_buf; }; enum sfxge_port_state { SFXGE_PORT_UNINITIALIZED = 0, SFXGE_PORT_INITIALIZED, SFXGE_PORT_STARTED }; struct sfxge_port { struct sfxge_softc *sc; struct mtx lock; enum sfxge_port_state init_state; #ifndef SFXGE_HAVE_PAUSE_MEDIAOPTS unsigned int wanted_fc; #endif struct sfxge_hw_stats phy_stats; struct sfxge_hw_stats mac_stats; efx_link_mode_t link_mode; uint8_t mcast_addrs[EFX_MAC_MULTICAST_LIST_MAX * EFX_MAC_ADDR_LEN]; unsigned int mcast_count; /* Only used in debugging output */ char lock_name[SFXGE_LOCK_NAME_MAX]; }; enum sfxge_softc_state { SFXGE_UNINITIALIZED = 0, SFXGE_INITIALIZED, SFXGE_REGISTERED, SFXGE_STARTED }; struct sfxge_softc { device_t dev; struct sx softc_lock; char softc_lock_name[SFXGE_LOCK_NAME_MAX]; enum sfxge_softc_state init_state; struct ifnet *ifnet; unsigned int if_flags; struct sysctl_oid *stats_node; struct sysctl_oid *txqs_node; struct task task_reset; efx_family_t family; caddr_t vpd_data; size_t vpd_size; efx_nic_t *enp; efsys_lock_t enp_lock; unsigned int rxq_entries; unsigned int txq_entries; bus_dma_tag_t parent_dma_tag; efsys_bar_t bar; struct sfxge_intr intr; struct sfxge_mcdi mcdi; struct sfxge_port port; uint32_t buffer_table_next; struct sfxge_evq *evq[SFXGE_RX_SCALE_MAX]; unsigned int ev_moderation; #if EFSYS_OPT_QSTATS clock_t ev_stats_update_time; uint64_t ev_stats[EV_NQSTATS]; #endif unsigned int max_rss_channels; uma_zone_t rxq_cache; struct sfxge_rxq *rxq[SFXGE_RX_SCALE_MAX]; unsigned int rx_indir_table[SFXGE_RX_SCALE_MAX]; struct sfxge_txq *txq[SFXGE_TXQ_NTYPES + SFXGE_RX_SCALE_MAX]; struct ifmedia media; size_t rx_prefix_size; size_t rx_buffer_size; size_t rx_buffer_align; uma_zone_t rx_buffer_zone; unsigned int evq_max; unsigned int evq_count; unsigned int rxq_count; unsigned int txq_count; - int tso_fw_assisted; + unsigned int tso_fw_assisted; +#define SFXGE_FATSOV1 (1 << 0) +#define SFXGE_FATSOV2 (1 << 1) + #if EFSYS_OPT_MCDI_LOGGING int mcdi_logging; #endif }; #define SFXGE_LINK_UP(sc) ((sc)->port.link_mode != EFX_LINK_DOWN) #define SFXGE_RUNNING(sc) ((sc)->ifnet->if_drv_flags & IFF_DRV_RUNNING) #define SFXGE_PARAM(_name) "hw.sfxge." #_name SYSCTL_DECL(_hw_sfxge); /* * From sfxge.c. */ extern void sfxge_schedule_reset(struct sfxge_softc *sc); extern void sfxge_sram_buf_tbl_alloc(struct sfxge_softc *sc, size_t n, uint32_t *idp); /* * From sfxge_dma.c. */ extern int sfxge_dma_init(struct sfxge_softc *sc); extern void sfxge_dma_fini(struct sfxge_softc *sc); extern int sfxge_dma_alloc(struct sfxge_softc *sc, bus_size_t len, efsys_mem_t *esmp); extern void sfxge_dma_free(efsys_mem_t *esmp); extern int sfxge_dma_map_sg_collapse(bus_dma_tag_t tag, bus_dmamap_t map, struct mbuf **mp, bus_dma_segment_t *segs, int *nsegs, int maxsegs); /* * From sfxge_ev.c. */ extern int sfxge_ev_init(struct sfxge_softc *sc); extern void sfxge_ev_fini(struct sfxge_softc *sc); extern int sfxge_ev_start(struct sfxge_softc *sc); extern void sfxge_ev_stop(struct sfxge_softc *sc); extern int sfxge_ev_qpoll(struct sfxge_evq *evq); /* * From sfxge_intr.c. */ extern int sfxge_intr_init(struct sfxge_softc *sc); extern void sfxge_intr_fini(struct sfxge_softc *sc); extern int sfxge_intr_start(struct sfxge_softc *sc); extern void sfxge_intr_stop(struct sfxge_softc *sc); /* * From sfxge_mcdi.c. */ extern int sfxge_mcdi_init(struct sfxge_softc *sc); extern void sfxge_mcdi_fini(struct sfxge_softc *sc); extern int sfxge_mcdi_ioctl(struct sfxge_softc *sc, sfxge_ioc_t *ip); /* * From sfxge_nvram.c. */ extern int sfxge_nvram_ioctl(struct sfxge_softc *sc, sfxge_ioc_t *ip); /* * From sfxge_port.c. */ extern int sfxge_port_init(struct sfxge_softc *sc); extern void sfxge_port_fini(struct sfxge_softc *sc); extern int sfxge_port_start(struct sfxge_softc *sc); extern void sfxge_port_stop(struct sfxge_softc *sc); extern void sfxge_mac_link_update(struct sfxge_softc *sc, efx_link_mode_t mode); extern int sfxge_mac_filter_set(struct sfxge_softc *sc); extern int sfxge_port_ifmedia_init(struct sfxge_softc *sc); extern uint64_t sfxge_get_counter(struct ifnet *ifp, ift_counter c); #define SFXGE_MAX_MTU (9 * 1024) #define SFXGE_ADAPTER_LOCK_INIT(_sc, _ifname) \ do { \ struct sfxge_softc *__sc = (_sc); \ \ snprintf((__sc)->softc_lock_name, \ sizeof((__sc)->softc_lock_name), \ "%s:softc", (_ifname)); \ sx_init(&(__sc)->softc_lock, (__sc)->softc_lock_name); \ } while (B_FALSE) #define SFXGE_ADAPTER_LOCK_DESTROY(_sc) \ sx_destroy(&(_sc)->softc_lock) #define SFXGE_ADAPTER_LOCK(_sc) \ sx_xlock(&(_sc)->softc_lock) #define SFXGE_ADAPTER_UNLOCK(_sc) \ sx_xunlock(&(_sc)->softc_lock) #define SFXGE_ADAPTER_LOCK_ASSERT_OWNED(_sc) \ sx_assert(&(_sc)->softc_lock, LA_XLOCKED) #define SFXGE_PORT_LOCK_INIT(_port, _ifname) \ do { \ struct sfxge_port *__port = (_port); \ \ snprintf((__port)->lock_name, \ sizeof((__port)->lock_name), \ "%s:port", (_ifname)); \ mtx_init(&(__port)->lock, (__port)->lock_name, \ NULL, MTX_DEF); \ } while (B_FALSE) #define SFXGE_PORT_LOCK_DESTROY(_port) \ mtx_destroy(&(_port)->lock) #define SFXGE_PORT_LOCK(_port) \ mtx_lock(&(_port)->lock) #define SFXGE_PORT_UNLOCK(_port) \ mtx_unlock(&(_port)->lock) #define SFXGE_PORT_LOCK_ASSERT_OWNED(_port) \ mtx_assert(&(_port)->lock, MA_OWNED) #define SFXGE_MCDI_LOCK_INIT(_mcdi, _ifname) \ do { \ struct sfxge_mcdi *__mcdi = (_mcdi); \ \ snprintf((__mcdi)->lock_name, \ sizeof((__mcdi)->lock_name), \ "%s:mcdi", (_ifname)); \ mtx_init(&(__mcdi)->lock, (__mcdi)->lock_name, \ NULL, MTX_DEF); \ } while (B_FALSE) #define SFXGE_MCDI_LOCK_DESTROY(_mcdi) \ mtx_destroy(&(_mcdi)->lock) #define SFXGE_MCDI_LOCK(_mcdi) \ mtx_lock(&(_mcdi)->lock) #define SFXGE_MCDI_UNLOCK(_mcdi) \ mtx_unlock(&(_mcdi)->lock) #define SFXGE_MCDI_LOCK_ASSERT_OWNED(_mcdi) \ mtx_assert(&(_mcdi)->lock, MA_OWNED) #define SFXGE_EVQ_LOCK_INIT(_evq, _ifname, _evq_index) \ do { \ struct sfxge_evq *__evq = (_evq); \ \ snprintf((__evq)->lock_name, \ sizeof((__evq)->lock_name), \ "%s:evq%u", (_ifname), (_evq_index)); \ mtx_init(&(__evq)->lock, (__evq)->lock_name, \ NULL, MTX_DEF); \ } while (B_FALSE) #define SFXGE_EVQ_LOCK_DESTROY(_evq) \ mtx_destroy(&(_evq)->lock) #define SFXGE_EVQ_LOCK(_evq) \ mtx_lock(&(_evq)->lock) #define SFXGE_EVQ_UNLOCK(_evq) \ mtx_unlock(&(_evq)->lock) #define SFXGE_EVQ_LOCK_ASSERT_OWNED(_evq) \ mtx_assert(&(_evq)->lock, MA_OWNED) #endif /* _SFXGE_H */ Index: head/sys/dev/sfxge/sfxge_tx.c =================================================================== --- head/sys/dev/sfxge/sfxge_tx.c (revision 294076) +++ head/sys/dev/sfxge/sfxge_tx.c (revision 294077) @@ -1,1897 +1,1987 @@ /*- * Copyright (c) 2010-2015 Solarflare Communications Inc. * All rights reserved. * * This software was developed in part by Philip Paeps under contract for * Solarflare Communications, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The views and conclusions contained in the software and documentation are * those of the authors and should not be interpreted as representing official * policies, either expressed or implied, of the FreeBSD Project. */ /* Theory of operation: * * Tx queues allocation and mapping * * One Tx queue with enabled checksum offload is allocated per Rx channel * (event queue). Also 2 Tx queues (one without checksum offload and one * with IP checksum offload only) are allocated and bound to event queue 0. * sfxge_txq_type is used as Tx queue label. * * So, event queue plus label mapping to Tx queue index is: * if event queue index is 0, TxQ-index = TxQ-label * [0..SFXGE_TXQ_NTYPES) * else TxQ-index = SFXGE_TXQ_NTYPES + EvQ-index - 1 * See sfxge_get_txq_by_label() sfxge_ev.c */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include +#include #include #include #include #include #include #include #include #include #include "common/efx.h" #include "sfxge.h" #include "sfxge_tx.h" #define SFXGE_PARAM_TX_DPL_GET_MAX SFXGE_PARAM(tx_dpl_get_max) static int sfxge_tx_dpl_get_max = SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT; TUNABLE_INT(SFXGE_PARAM_TX_DPL_GET_MAX, &sfxge_tx_dpl_get_max); SYSCTL_INT(_hw_sfxge, OID_AUTO, tx_dpl_get_max, CTLFLAG_RDTUN, &sfxge_tx_dpl_get_max, 0, "Maximum number of any packets in deferred packet get-list"); #define SFXGE_PARAM_TX_DPL_GET_NON_TCP_MAX \ SFXGE_PARAM(tx_dpl_get_non_tcp_max) static int sfxge_tx_dpl_get_non_tcp_max = SFXGE_TX_DPL_GET_NON_TCP_PKT_LIMIT_DEFAULT; TUNABLE_INT(SFXGE_PARAM_TX_DPL_GET_NON_TCP_MAX, &sfxge_tx_dpl_get_non_tcp_max); SYSCTL_INT(_hw_sfxge, OID_AUTO, tx_dpl_get_non_tcp_max, CTLFLAG_RDTUN, &sfxge_tx_dpl_get_non_tcp_max, 0, "Maximum number of non-TCP packets in deferred packet get-list"); #define SFXGE_PARAM_TX_DPL_PUT_MAX SFXGE_PARAM(tx_dpl_put_max) static int sfxge_tx_dpl_put_max = SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT; TUNABLE_INT(SFXGE_PARAM_TX_DPL_PUT_MAX, &sfxge_tx_dpl_put_max); SYSCTL_INT(_hw_sfxge, OID_AUTO, tx_dpl_put_max, CTLFLAG_RDTUN, &sfxge_tx_dpl_put_max, 0, "Maximum number of any packets in deferred packet put-list"); #define SFXGE_PARAM_TSO_FW_ASSISTED SFXGE_PARAM(tso_fw_assisted) -static int sfxge_tso_fw_assisted = 1; +static int sfxge_tso_fw_assisted = (SFXGE_FATSOV1 | SFXGE_FATSOV2); TUNABLE_INT(SFXGE_PARAM_TSO_FW_ASSISTED, &sfxge_tso_fw_assisted); SYSCTL_INT(_hw_sfxge, OID_AUTO, tso_fw_assisted, CTLFLAG_RDTUN, &sfxge_tso_fw_assisted, 0, - "Use FW-assisted TSO if supported by NIC firmware"); + "Bitmask of FW-assisted TSO allowed to use if supported by NIC firmware"); static const struct { const char *name; size_t offset; } sfxge_tx_stats[] = { #define SFXGE_TX_STAT(name, member) \ { #name, offsetof(struct sfxge_txq, member) } SFXGE_TX_STAT(tso_bursts, tso_bursts), SFXGE_TX_STAT(tso_packets, tso_packets), SFXGE_TX_STAT(tso_long_headers, tso_long_headers), SFXGE_TX_STAT(tso_pdrop_too_many, tso_pdrop_too_many), SFXGE_TX_STAT(tso_pdrop_no_rsrc, tso_pdrop_no_rsrc), SFXGE_TX_STAT(tx_collapses, collapses), SFXGE_TX_STAT(tx_drops, drops), SFXGE_TX_STAT(tx_get_overflow, get_overflow), SFXGE_TX_STAT(tx_get_non_tcp_overflow, get_non_tcp_overflow), SFXGE_TX_STAT(tx_put_overflow, put_overflow), SFXGE_TX_STAT(tx_netdown_drops, netdown_drops), }; /* Forward declarations. */ static void sfxge_tx_qdpl_service(struct sfxge_txq *txq); static void sfxge_tx_qlist_post(struct sfxge_txq *txq); static void sfxge_tx_qunblock(struct sfxge_txq *txq); static int sfxge_tx_queue_tso(struct sfxge_txq *txq, struct mbuf *mbuf, const bus_dma_segment_t *dma_seg, int n_dma_seg, int vlan_tagged); static int sfxge_tx_maybe_insert_tag(struct sfxge_txq *txq, struct mbuf *mbuf) { uint16_t this_tag = ((mbuf->m_flags & M_VLANTAG) ? mbuf->m_pkthdr.ether_vtag : 0); if (this_tag == txq->hw_vlan_tci) return (0); efx_tx_qdesc_vlantci_create(txq->common, bswap16(this_tag), &txq->pend_desc[0]); txq->n_pend_desc = 1; txq->hw_vlan_tci = this_tag; return (1); } static inline void sfxge_next_stmp(struct sfxge_txq *txq, struct sfxge_tx_mapping **pstmp) { KASSERT((*pstmp)->flags == 0, ("stmp flags are not 0")); if (__predict_false(*pstmp == &txq->stmp[txq->ptr_mask])) *pstmp = &txq->stmp[0]; else (*pstmp)++; } void sfxge_tx_qcomplete(struct sfxge_txq *txq, struct sfxge_evq *evq) { unsigned int completed; SFXGE_EVQ_LOCK_ASSERT_OWNED(evq); completed = txq->completed; while (completed != txq->pending) { struct sfxge_tx_mapping *stmp; unsigned int id; id = completed++ & txq->ptr_mask; stmp = &txq->stmp[id]; if (stmp->flags & TX_BUF_UNMAP) { bus_dmamap_unload(txq->packet_dma_tag, stmp->map); if (stmp->flags & TX_BUF_MBUF) { struct mbuf *m = stmp->u.mbuf; do m = m_free(m); while (m != NULL); } else { free(stmp->u.heap_buf, M_SFXGE); } stmp->flags = 0; } } txq->completed = completed; /* Check whether we need to unblock the queue. */ mb(); if (txq->blocked) { unsigned int level; level = txq->added - txq->completed; if (level <= SFXGE_TXQ_UNBLOCK_LEVEL(txq->entries)) sfxge_tx_qunblock(txq); } } static unsigned int sfxge_is_mbuf_non_tcp(struct mbuf *mbuf) { /* Absense of TCP checksum flags does not mean that it is non-TCP * but it should be true if user wants to achieve high throughput. */ return (!(mbuf->m_pkthdr.csum_flags & (CSUM_IP_TCP | CSUM_IP6_TCP))); } /* * Reorder the put list and append it to the get list. */ static void sfxge_tx_qdpl_swizzle(struct sfxge_txq *txq) { struct sfxge_tx_dpl *stdp; struct mbuf *mbuf, *get_next, **get_tailp; volatile uintptr_t *putp; uintptr_t put; unsigned int count; unsigned int non_tcp_count; SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); stdp = &txq->dpl; /* Acquire the put list. */ putp = &stdp->std_put; put = atomic_readandclear_ptr(putp); mbuf = (void *)put; if (mbuf == NULL) return; /* Reverse the put list. */ get_tailp = &mbuf->m_nextpkt; get_next = NULL; count = 0; non_tcp_count = 0; do { struct mbuf *put_next; non_tcp_count += sfxge_is_mbuf_non_tcp(mbuf); put_next = mbuf->m_nextpkt; mbuf->m_nextpkt = get_next; get_next = mbuf; mbuf = put_next; count++; } while (mbuf != NULL); if (count > stdp->std_put_hiwat) stdp->std_put_hiwat = count; /* Append the reversed put list to the get list. */ KASSERT(*get_tailp == NULL, ("*get_tailp != NULL")); *stdp->std_getp = get_next; stdp->std_getp = get_tailp; stdp->std_get_count += count; stdp->std_get_non_tcp_count += non_tcp_count; } static void sfxge_tx_qreap(struct sfxge_txq *txq) { SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); txq->reaped = txq->completed; } static void sfxge_tx_qlist_post(struct sfxge_txq *txq) { unsigned int old_added; unsigned int block_level; unsigned int level; int rc; SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); KASSERT(txq->n_pend_desc != 0, ("txq->n_pend_desc == 0")); KASSERT(txq->n_pend_desc <= txq->max_pkt_desc, ("txq->n_pend_desc too large")); KASSERT(!txq->blocked, ("txq->blocked")); old_added = txq->added; /* Post the fragment list. */ rc = efx_tx_qdesc_post(txq->common, txq->pend_desc, txq->n_pend_desc, txq->reaped, &txq->added); KASSERT(rc == 0, ("efx_tx_qdesc_post() failed")); /* If efx_tx_qdesc_post() had to refragment, our information about * buffers to free may be associated with the wrong * descriptors. */ KASSERT(txq->added - old_added == txq->n_pend_desc, ("efx_tx_qdesc_post() refragmented descriptors")); level = txq->added - txq->reaped; KASSERT(level <= txq->entries, ("overfilled TX queue")); /* Clear the fragment list. */ txq->n_pend_desc = 0; /* * Set the block level to ensure there is space to generate a * large number of descriptors for TSO. */ block_level = EFX_TXQ_LIMIT(txq->entries) - txq->max_pkt_desc; /* Have we reached the block level? */ if (level < block_level) return; /* Reap, and check again */ sfxge_tx_qreap(txq); level = txq->added - txq->reaped; if (level < block_level) return; txq->blocked = 1; /* * Avoid a race with completion interrupt handling that could leave * the queue blocked. */ mb(); sfxge_tx_qreap(txq); level = txq->added - txq->reaped; if (level < block_level) { mb(); txq->blocked = 0; } } static int sfxge_tx_queue_mbuf(struct sfxge_txq *txq, struct mbuf *mbuf) { bus_dmamap_t *used_map; bus_dmamap_t map; bus_dma_segment_t dma_seg[SFXGE_TX_MAPPING_MAX_SEG]; unsigned int id; struct sfxge_tx_mapping *stmp; efx_desc_t *desc; int n_dma_seg; int rc; int i; int eop; int vlan_tagged; KASSERT(!txq->blocked, ("txq->blocked")); if (mbuf->m_pkthdr.csum_flags & CSUM_TSO) prefetch_read_many(mbuf->m_data); if (__predict_false(txq->init_state != SFXGE_TXQ_STARTED)) { rc = EINTR; goto reject; } /* Load the packet for DMA. */ id = txq->added & txq->ptr_mask; stmp = &txq->stmp[id]; rc = bus_dmamap_load_mbuf_sg(txq->packet_dma_tag, stmp->map, mbuf, dma_seg, &n_dma_seg, 0); if (rc == EFBIG) { /* Try again. */ struct mbuf *new_mbuf = m_collapse(mbuf, M_NOWAIT, SFXGE_TX_MAPPING_MAX_SEG); if (new_mbuf == NULL) goto reject; ++txq->collapses; mbuf = new_mbuf; rc = bus_dmamap_load_mbuf_sg(txq->packet_dma_tag, stmp->map, mbuf, dma_seg, &n_dma_seg, 0); } if (rc != 0) goto reject; /* Make the packet visible to the hardware. */ bus_dmamap_sync(txq->packet_dma_tag, stmp->map, BUS_DMASYNC_PREWRITE); used_map = &stmp->map; vlan_tagged = sfxge_tx_maybe_insert_tag(txq, mbuf); if (vlan_tagged) { sfxge_next_stmp(txq, &stmp); } if (mbuf->m_pkthdr.csum_flags & CSUM_TSO) { rc = sfxge_tx_queue_tso(txq, mbuf, dma_seg, n_dma_seg, vlan_tagged); if (rc < 0) goto reject_mapped; stmp = &txq->stmp[(rc - 1) & txq->ptr_mask]; } else { /* Add the mapping to the fragment list, and set flags * for the buffer. */ i = 0; for (;;) { desc = &txq->pend_desc[i + vlan_tagged]; eop = (i == n_dma_seg - 1); efx_tx_qdesc_dma_create(txq->common, dma_seg[i].ds_addr, dma_seg[i].ds_len, eop, desc); if (eop) break; i++; sfxge_next_stmp(txq, &stmp); } txq->n_pend_desc = n_dma_seg + vlan_tagged; } /* * If the mapping required more than one descriptor * then we need to associate the DMA map with the last * descriptor, not the first. */ if (used_map != &stmp->map) { map = stmp->map; stmp->map = *used_map; *used_map = map; } stmp->u.mbuf = mbuf; stmp->flags = TX_BUF_UNMAP | TX_BUF_MBUF; /* Post the fragment list. */ sfxge_tx_qlist_post(txq); return (0); reject_mapped: bus_dmamap_unload(txq->packet_dma_tag, *used_map); reject: /* Drop the packet on the floor. */ m_freem(mbuf); ++txq->drops; return (rc); } /* * Drain the deferred packet list into the transmit queue. */ static void sfxge_tx_qdpl_drain(struct sfxge_txq *txq) { struct sfxge_softc *sc; struct sfxge_tx_dpl *stdp; struct mbuf *mbuf, *next; unsigned int count; unsigned int non_tcp_count; unsigned int pushed; int rc; SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); sc = txq->sc; stdp = &txq->dpl; pushed = txq->added; if (__predict_true(txq->init_state == SFXGE_TXQ_STARTED)) { prefetch_read_many(sc->enp); prefetch_read_many(txq->common); } mbuf = stdp->std_get; count = stdp->std_get_count; non_tcp_count = stdp->std_get_non_tcp_count; if (count > stdp->std_get_hiwat) stdp->std_get_hiwat = count; while (count != 0) { KASSERT(mbuf != NULL, ("mbuf == NULL")); next = mbuf->m_nextpkt; mbuf->m_nextpkt = NULL; ETHER_BPF_MTAP(sc->ifnet, mbuf); /* packet capture */ if (next != NULL) prefetch_read_many(next); rc = sfxge_tx_queue_mbuf(txq, mbuf); --count; non_tcp_count -= sfxge_is_mbuf_non_tcp(mbuf); mbuf = next; if (rc != 0) continue; if (txq->blocked) break; /* Push the fragments to the hardware in batches. */ if (txq->added - pushed >= SFXGE_TX_BATCH) { efx_tx_qpush(txq->common, txq->added, pushed); pushed = txq->added; } } if (count == 0) { KASSERT(mbuf == NULL, ("mbuf != NULL")); KASSERT(non_tcp_count == 0, ("inconsistent TCP/non-TCP detection")); stdp->std_get = NULL; stdp->std_get_count = 0; stdp->std_get_non_tcp_count = 0; stdp->std_getp = &stdp->std_get; } else { stdp->std_get = mbuf; stdp->std_get_count = count; stdp->std_get_non_tcp_count = non_tcp_count; } if (txq->added != pushed) efx_tx_qpush(txq->common, txq->added, pushed); KASSERT(txq->blocked || stdp->std_get_count == 0, ("queue unblocked but count is non-zero")); } #define SFXGE_TX_QDPL_PENDING(_txq) ((_txq)->dpl.std_put != 0) /* * Service the deferred packet list. * * NOTE: drops the txq mutex! */ static void sfxge_tx_qdpl_service(struct sfxge_txq *txq) { SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); do { if (SFXGE_TX_QDPL_PENDING(txq)) sfxge_tx_qdpl_swizzle(txq); if (!txq->blocked) sfxge_tx_qdpl_drain(txq); SFXGE_TXQ_UNLOCK(txq); } while (SFXGE_TX_QDPL_PENDING(txq) && SFXGE_TXQ_TRYLOCK(txq)); } /* * Put a packet on the deferred packet get-list. */ static int sfxge_tx_qdpl_put_locked(struct sfxge_txq *txq, struct mbuf *mbuf) { struct sfxge_tx_dpl *stdp; stdp = &txq->dpl; KASSERT(mbuf->m_nextpkt == NULL, ("mbuf->m_nextpkt != NULL")); SFXGE_TXQ_LOCK_ASSERT_OWNED(txq); if (stdp->std_get_count >= stdp->std_get_max) { txq->get_overflow++; return (ENOBUFS); } if (sfxge_is_mbuf_non_tcp(mbuf)) { if (stdp->std_get_non_tcp_count >= stdp->std_get_non_tcp_max) { txq->get_non_tcp_overflow++; return (ENOBUFS); } stdp->std_get_non_tcp_count++; } *(stdp->std_getp) = mbuf; stdp->std_getp = &mbuf->m_nextpkt; stdp->std_get_count++; return (0); } /* * Put a packet on the deferred packet put-list. * * We overload the csum_data field in the mbuf to keep track of this length * because there is no cheap alternative to avoid races. */ static int sfxge_tx_qdpl_put_unlocked(struct sfxge_txq *txq, struct mbuf *mbuf) { struct sfxge_tx_dpl *stdp; volatile uintptr_t *putp; uintptr_t old; uintptr_t new; unsigned old_len; KASSERT(mbuf->m_nextpkt == NULL, ("mbuf->m_nextpkt != NULL")); SFXGE_TXQ_LOCK_ASSERT_NOTOWNED(txq); stdp = &txq->dpl; putp = &stdp->std_put; new = (uintptr_t)mbuf; do { old = *putp; if (old != 0) { struct mbuf *mp = (struct mbuf *)old; old_len = mp->m_pkthdr.csum_data; } else old_len = 0; if (old_len >= stdp->std_put_max) { atomic_add_long(&txq->put_overflow, 1); return (ENOBUFS); } mbuf->m_pkthdr.csum_data = old_len + 1; mbuf->m_nextpkt = (void *)old; } while (atomic_cmpset_ptr(putp, old, new) == 0); return (0); } /* * Called from if_transmit - will try to grab the txq lock and enqueue to the * put list if it succeeds, otherwise try to push onto the defer list if space. */ static int sfxge_tx_packet_add(struct sfxge_txq *txq, struct mbuf *m) { int rc; if (!SFXGE_LINK_UP(txq->sc)) { atomic_add_long(&txq->netdown_drops, 1); return (ENETDOWN); } /* * Try to grab the txq lock. If we are able to get the lock, * the packet will be appended to the "get list" of the deferred * packet list. Otherwise, it will be pushed on the "put list". */ if (SFXGE_TXQ_TRYLOCK(txq)) { /* First swizzle put-list to get-list to keep order */ sfxge_tx_qdpl_swizzle(txq); rc = sfxge_tx_qdpl_put_locked(txq, m); /* Try to service the list. */ sfxge_tx_qdpl_service(txq); /* Lock has been dropped. */ } else { rc = sfxge_tx_qdpl_put_unlocked(txq, m); /* * Try to grab the lock again. * * If we are able to get the lock, we need to process * the deferred packet list. If we are not able to get * the lock, another thread is processing the list. */ if ((rc == 0) && SFXGE_TXQ_TRYLOCK(txq)) { sfxge_tx_qdpl_service(txq); /* Lock has been dropped. */ } } SFXGE_TXQ_LOCK_ASSERT_NOTOWNED(txq); return (rc); } static void sfxge_tx_qdpl_flush(struct sfxge_txq *txq) { struct sfxge_tx_dpl *stdp = &txq->dpl; struct mbuf *mbuf, *next; SFXGE_TXQ_LOCK(txq); sfxge_tx_qdpl_swizzle(txq); for (mbuf = stdp->std_get; mbuf != NULL; mbuf = next) { next = mbuf->m_nextpkt; m_freem(mbuf); } stdp->std_get = NULL; stdp->std_get_count = 0; stdp->std_get_non_tcp_count = 0; stdp->std_getp = &stdp->std_get; SFXGE_TXQ_UNLOCK(txq); } void sfxge_if_qflush(struct ifnet *ifp) { struct sfxge_softc *sc; unsigned int i; sc = ifp->if_softc; for (i = 0; i < sc->txq_count; i++) sfxge_tx_qdpl_flush(sc->txq[i]); } #if SFXGE_TX_PARSE_EARLY /* There is little space for user data in mbuf pkthdr, so we * use l*hlen fields which are not used by the driver otherwise * to store header offsets. * The fields are 8-bit, but it's ok, no header may be longer than 255 bytes. */ #define TSO_MBUF_PROTO(_mbuf) ((_mbuf)->m_pkthdr.PH_loc.sixteen[0]) /* We abuse l5hlen here because PH_loc can hold only 64 bits of data */ #define TSO_MBUF_FLAGS(_mbuf) ((_mbuf)->m_pkthdr.l5hlen) #define TSO_MBUF_PACKETID(_mbuf) ((_mbuf)->m_pkthdr.PH_loc.sixteen[1]) #define TSO_MBUF_SEQNUM(_mbuf) ((_mbuf)->m_pkthdr.PH_loc.thirtytwo[1]) static void sfxge_parse_tx_packet(struct mbuf *mbuf) { struct ether_header *eh = mtod(mbuf, struct ether_header *); const struct tcphdr *th; struct tcphdr th_copy; /* Find network protocol and header */ TSO_MBUF_PROTO(mbuf) = eh->ether_type; if (TSO_MBUF_PROTO(mbuf) == htons(ETHERTYPE_VLAN)) { struct ether_vlan_header *veh = mtod(mbuf, struct ether_vlan_header *); TSO_MBUF_PROTO(mbuf) = veh->evl_proto; mbuf->m_pkthdr.l2hlen = sizeof(*veh); } else { mbuf->m_pkthdr.l2hlen = sizeof(*eh); } /* Find TCP header */ if (TSO_MBUF_PROTO(mbuf) == htons(ETHERTYPE_IP)) { const struct ip *iph = (const struct ip *)mtodo(mbuf, mbuf->m_pkthdr.l2hlen); KASSERT(iph->ip_p == IPPROTO_TCP, ("TSO required on non-TCP packet")); mbuf->m_pkthdr.l3hlen = mbuf->m_pkthdr.l2hlen + 4 * iph->ip_hl; TSO_MBUF_PACKETID(mbuf) = iph->ip_id; } else { KASSERT(TSO_MBUF_PROTO(mbuf) == htons(ETHERTYPE_IPV6), ("TSO required on non-IP packet")); KASSERT(((const struct ip6_hdr *)mtodo(mbuf, mbuf->m_pkthdr.l2hlen))->ip6_nxt == IPPROTO_TCP, ("TSO required on non-TCP packet")); mbuf->m_pkthdr.l3hlen = mbuf->m_pkthdr.l2hlen + sizeof(struct ip6_hdr); TSO_MBUF_PACKETID(mbuf) = 0; } KASSERT(mbuf->m_len >= mbuf->m_pkthdr.l3hlen, ("network header is fragmented in mbuf")); /* We need TCP header including flags (window is the next) */ if (mbuf->m_len < mbuf->m_pkthdr.l3hlen + offsetof(struct tcphdr, th_win)) { m_copydata(mbuf, mbuf->m_pkthdr.l3hlen, sizeof(th_copy), (caddr_t)&th_copy); th = &th_copy; } else { th = (const struct tcphdr *)mtodo(mbuf, mbuf->m_pkthdr.l3hlen); } mbuf->m_pkthdr.l4hlen = mbuf->m_pkthdr.l3hlen + 4 * th->th_off; TSO_MBUF_SEQNUM(mbuf) = ntohl(th->th_seq); /* These flags must not be duplicated */ /* * RST should not be duplicated as well, but FreeBSD kernel * generates TSO packets with RST flag. So, do not assert * its absence. */ KASSERT(!(th->th_flags & (TH_URG | TH_SYN)), ("incompatible TCP flag 0x%x on TSO packet", th->th_flags & (TH_URG | TH_SYN))); TSO_MBUF_FLAGS(mbuf) = th->th_flags; } #endif /* * TX start -- called by the stack. */ int sfxge_if_transmit(struct ifnet *ifp, struct mbuf *m) { struct sfxge_softc *sc; struct sfxge_txq *txq; int rc; sc = (struct sfxge_softc *)ifp->if_softc; /* * Transmit may be called when interface is up from the kernel * point of view, but not yet up (in progress) from the driver * point of view. I.e. link aggregation bring up. * Transmit may be called when interface is up from the driver * point of view, but already down from the kernel point of * view. I.e. Rx when interface shutdown is in progress. */ KASSERT((ifp->if_flags & IFF_UP) || (sc->if_flags & IFF_UP), ("interface not up")); /* Pick the desired transmit queue. */ if (m->m_pkthdr.csum_flags & (CSUM_DELAY_DATA | CSUM_TCP_IPV6 | CSUM_UDP_IPV6 | CSUM_TSO)) { int index = 0; /* check if flowid is set */ if (M_HASHTYPE_GET(m) != M_HASHTYPE_NONE) { uint32_t hash = m->m_pkthdr.flowid; index = sc->rx_indir_table[hash % SFXGE_RX_SCALE_MAX]; } #if SFXGE_TX_PARSE_EARLY if (m->m_pkthdr.csum_flags & CSUM_TSO) sfxge_parse_tx_packet(m); #endif txq = sc->txq[SFXGE_TXQ_IP_TCP_UDP_CKSUM + index]; } else if (m->m_pkthdr.csum_flags & CSUM_DELAY_IP) { txq = sc->txq[SFXGE_TXQ_IP_CKSUM]; } else { txq = sc->txq[SFXGE_TXQ_NON_CKSUM]; } rc = sfxge_tx_packet_add(txq, m); if (rc != 0) m_freem(m); return (rc); } /* * Software "TSO". Not quite as good as doing it in hardware, but * still faster than segmenting in the stack. */ struct sfxge_tso_state { /* Output position */ unsigned out_len; /* Remaining length in current segment */ unsigned seqnum; /* Current sequence number */ unsigned packet_space; /* Remaining space in current packet */ + unsigned segs_space; /* Remaining number of DMA segments + for the packet (FATSOv2 only) */ /* Input position */ uint64_t dma_addr; /* DMA address of current position */ unsigned in_len; /* Remaining length in current mbuf */ const struct mbuf *mbuf; /* Input mbuf (head of chain) */ u_short protocol; /* Network protocol (after VLAN decap) */ ssize_t nh_off; /* Offset of network header */ ssize_t tcph_off; /* Offset of TCP header */ unsigned header_len; /* Number of bytes of header */ unsigned seg_size; /* TCP segment size */ int fw_assisted; /* Use FW-assisted TSO */ u_short packet_id; /* IPv4 packet ID from the original packet */ uint8_t tcp_flags; /* TCP flags */ efx_desc_t header_desc; /* Precomputed header descriptor for * FW-assisted TSO */ }; #if !SFXGE_TX_PARSE_EARLY static const struct ip *tso_iph(const struct sfxge_tso_state *tso) { KASSERT(tso->protocol == htons(ETHERTYPE_IP), ("tso_iph() in non-IPv4 state")); return (const struct ip *)(tso->mbuf->m_data + tso->nh_off); } static __unused const struct ip6_hdr *tso_ip6h(const struct sfxge_tso_state *tso) { KASSERT(tso->protocol == htons(ETHERTYPE_IPV6), ("tso_ip6h() in non-IPv6 state")); return (const struct ip6_hdr *)(tso->mbuf->m_data + tso->nh_off); } static const struct tcphdr *tso_tcph(const struct sfxge_tso_state *tso) { return (const struct tcphdr *)(tso->mbuf->m_data + tso->tcph_off); } #endif /* Size of preallocated TSO header buffers. Larger blocks must be * allocated from the heap. */ #define TSOH_STD_SIZE 128 /* At most half the descriptors in the queue at any time will refer to * a TSO header buffer, since they must always be followed by a * payload descriptor referring to an mbuf. */ #define TSOH_COUNT(_txq_entries) ((_txq_entries) / 2u) #define TSOH_PER_PAGE (PAGE_SIZE / TSOH_STD_SIZE) #define TSOH_PAGE_COUNT(_txq_entries) \ ((TSOH_COUNT(_txq_entries) + TSOH_PER_PAGE - 1) / TSOH_PER_PAGE) static int tso_init(struct sfxge_txq *txq) { struct sfxge_softc *sc = txq->sc; unsigned int tsoh_page_count = TSOH_PAGE_COUNT(sc->txq_entries); int i, rc; /* Allocate TSO header buffers */ txq->tsoh_buffer = malloc(tsoh_page_count * sizeof(txq->tsoh_buffer[0]), M_SFXGE, M_WAITOK); for (i = 0; i < tsoh_page_count; i++) { rc = sfxge_dma_alloc(sc, PAGE_SIZE, &txq->tsoh_buffer[i]); if (rc != 0) goto fail; } return (0); fail: while (i-- > 0) sfxge_dma_free(&txq->tsoh_buffer[i]); free(txq->tsoh_buffer, M_SFXGE); txq->tsoh_buffer = NULL; return (rc); } static void tso_fini(struct sfxge_txq *txq) { int i; if (txq->tsoh_buffer != NULL) { for (i = 0; i < TSOH_PAGE_COUNT(txq->sc->txq_entries); i++) sfxge_dma_free(&txq->tsoh_buffer[i]); free(txq->tsoh_buffer, M_SFXGE); } } static void tso_start(struct sfxge_txq *txq, struct sfxge_tso_state *tso, const bus_dma_segment_t *hdr_dma_seg, struct mbuf *mbuf) { const efx_nic_cfg_t *encp = efx_nic_cfg_get(txq->sc->enp); #if !SFXGE_TX_PARSE_EARLY struct ether_header *eh = mtod(mbuf, struct ether_header *); const struct tcphdr *th; struct tcphdr th_copy; #endif - tso->fw_assisted = txq->sc->tso_fw_assisted; + tso->fw_assisted = txq->tso_fw_assisted; tso->mbuf = mbuf; /* Find network protocol and header */ #if !SFXGE_TX_PARSE_EARLY tso->protocol = eh->ether_type; if (tso->protocol == htons(ETHERTYPE_VLAN)) { struct ether_vlan_header *veh = mtod(mbuf, struct ether_vlan_header *); tso->protocol = veh->evl_proto; tso->nh_off = sizeof(*veh); } else { tso->nh_off = sizeof(*eh); } #else tso->protocol = TSO_MBUF_PROTO(mbuf); tso->nh_off = mbuf->m_pkthdr.l2hlen; tso->tcph_off = mbuf->m_pkthdr.l3hlen; tso->packet_id = TSO_MBUF_PACKETID(mbuf); #endif #if !SFXGE_TX_PARSE_EARLY /* Find TCP header */ if (tso->protocol == htons(ETHERTYPE_IP)) { KASSERT(tso_iph(tso)->ip_p == IPPROTO_TCP, ("TSO required on non-TCP packet")); tso->tcph_off = tso->nh_off + 4 * tso_iph(tso)->ip_hl; tso->packet_id = tso_iph(tso)->ip_id; } else { KASSERT(tso->protocol == htons(ETHERTYPE_IPV6), ("TSO required on non-IP packet")); KASSERT(tso_ip6h(tso)->ip6_nxt == IPPROTO_TCP, ("TSO required on non-TCP packet")); tso->tcph_off = tso->nh_off + sizeof(struct ip6_hdr); tso->packet_id = 0; } #endif if (tso->fw_assisted && __predict_false(tso->tcph_off > encp->enc_tx_tso_tcp_header_offset_limit)) { tso->fw_assisted = 0; } #if !SFXGE_TX_PARSE_EARLY KASSERT(mbuf->m_len >= tso->tcph_off, ("network header is fragmented in mbuf")); /* We need TCP header including flags (window is the next) */ if (mbuf->m_len < tso->tcph_off + offsetof(struct tcphdr, th_win)) { m_copydata(tso->mbuf, tso->tcph_off, sizeof(th_copy), (caddr_t)&th_copy); th = &th_copy; } else { th = tso_tcph(tso); } tso->header_len = tso->tcph_off + 4 * th->th_off; #else tso->header_len = mbuf->m_pkthdr.l4hlen; #endif tso->seg_size = mbuf->m_pkthdr.tso_segsz; #if !SFXGE_TX_PARSE_EARLY tso->seqnum = ntohl(th->th_seq); /* These flags must not be duplicated */ /* * RST should not be duplicated as well, but FreeBSD kernel * generates TSO packets with RST flag. So, do not assert * its absence. */ KASSERT(!(th->th_flags & (TH_URG | TH_SYN)), ("incompatible TCP flag 0x%x on TSO packet", th->th_flags & (TH_URG | TH_SYN))); tso->tcp_flags = th->th_flags; #else tso->seqnum = TSO_MBUF_SEQNUM(mbuf); tso->tcp_flags = TSO_MBUF_FLAGS(mbuf); #endif tso->out_len = mbuf->m_pkthdr.len - tso->header_len; if (tso->fw_assisted) { if (hdr_dma_seg->ds_len >= tso->header_len) efx_tx_qdesc_dma_create(txq->common, hdr_dma_seg->ds_addr, tso->header_len, B_FALSE, &tso->header_desc); else tso->fw_assisted = 0; } } /* * tso_fill_packet_with_fragment - form descriptors for the current fragment * * Form descriptors for the current fragment, until we reach the end * of fragment or end-of-packet. Return 0 on success, 1 if not enough * space. */ static void tso_fill_packet_with_fragment(struct sfxge_txq *txq, struct sfxge_tso_state *tso) { efx_desc_t *desc; int n; + uint64_t dma_addr = tso->dma_addr; + boolean_t eop; if (tso->in_len == 0 || tso->packet_space == 0) return; KASSERT(tso->in_len > 0, ("TSO input length went negative")); KASSERT(tso->packet_space > 0, ("TSO packet space went negative")); - n = min(tso->in_len, tso->packet_space); + if (tso->fw_assisted & SFXGE_FATSOV2) { + n = tso->in_len; + tso->out_len -= n; + tso->seqnum += n; + tso->in_len = 0; + if (n < tso->packet_space) { + tso->packet_space -= n; + tso->segs_space--; + } else { + tso->packet_space = tso->seg_size - + (n - tso->packet_space) % tso->seg_size; + tso->segs_space = + EFX_TX_FATSOV2_DMA_SEGS_PER_PKT_MAX - 1 - + (tso->packet_space != tso->seg_size); + } + } else { + n = min(tso->in_len, tso->packet_space); + tso->packet_space -= n; + tso->out_len -= n; + tso->dma_addr += n; + tso->in_len -= n; + } - tso->packet_space -= n; - tso->out_len -= n; - tso->in_len -= n; + /* + * It is OK to use binary OR below to avoid extra branching + * since all conditions may always be checked. + */ + eop = (tso->out_len == 0) | (tso->packet_space == 0) | + (tso->segs_space == 0); desc = &txq->pend_desc[txq->n_pend_desc++]; - efx_tx_qdesc_dma_create(txq->common, - tso->dma_addr, - n, - tso->out_len == 0 || tso->packet_space == 0, - desc); - - tso->dma_addr += n; + efx_tx_qdesc_dma_create(txq->common, dma_addr, n, eop, desc); } /* Callback from bus_dmamap_load() for long TSO headers. */ static void tso_map_long_header(void *dma_addr_ret, bus_dma_segment_t *segs, int nseg, int error) { *(uint64_t *)dma_addr_ret = ((__predict_true(error == 0) && __predict_true(nseg == 1)) ? segs->ds_addr : 0); } /* * tso_start_new_packet - generate a new header and prepare for the new packet * * Generate a new header and prepare for the new packet. Return 0 on * success, or an error code if failed to alloc header. */ static int tso_start_new_packet(struct sfxge_txq *txq, struct sfxge_tso_state *tso, unsigned int *idp) { unsigned int id = *idp; struct tcphdr *tsoh_th; unsigned ip_length; caddr_t header; uint64_t dma_addr; bus_dmamap_t map; efx_desc_t *desc; int rc; if (tso->fw_assisted) { - uint8_t tcp_flags = tso->tcp_flags; + if (tso->fw_assisted & SFXGE_FATSOV2) { + /* Add 2 FATSOv2 option descriptors */ + desc = &txq->pend_desc[txq->n_pend_desc]; + efx_tx_qdesc_tso2_create(txq->common, + tso->packet_id, + tso->seqnum, + tso->seg_size, + desc, + EFX_TX_FATSOV2_OPT_NDESCS); + desc += EFX_TX_FATSOV2_OPT_NDESCS; + txq->n_pend_desc += EFX_TX_FATSOV2_OPT_NDESCS; + KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); + id = (id + EFX_TX_FATSOV2_OPT_NDESCS) & txq->ptr_mask; - if (tso->out_len > tso->seg_size) - tcp_flags &= ~(TH_FIN | TH_PUSH); + tso->segs_space = + EFX_TX_FATSOV2_DMA_SEGS_PER_PKT_MAX - 1; + } else { + uint8_t tcp_flags = tso->tcp_flags; - /* TSO option descriptor */ - desc = &txq->pend_desc[txq->n_pend_desc++]; - efx_tx_qdesc_tso_create(txq->common, - tso->packet_id, - tso->seqnum, - tcp_flags, - desc++); - KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); - id = (id + 1) & txq->ptr_mask; + if (tso->out_len > tso->seg_size) + tcp_flags &= ~(TH_FIN | TH_PUSH); + /* Add FATSOv1 option descriptor */ + desc = &txq->pend_desc[txq->n_pend_desc++]; + efx_tx_qdesc_tso_create(txq->common, + tso->packet_id, + tso->seqnum, + tcp_flags, + desc++); + KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); + id = (id + 1) & txq->ptr_mask; + + tso->seqnum += tso->seg_size; + tso->segs_space = UINT_MAX; + } + /* Header DMA descriptor */ *desc = tso->header_desc; txq->n_pend_desc++; KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); id = (id + 1) & txq->ptr_mask; - - tso->seqnum += tso->seg_size; } else { /* Allocate a DMA-mapped header buffer. */ if (__predict_true(tso->header_len <= TSOH_STD_SIZE)) { unsigned int page_index = (id / 2) / TSOH_PER_PAGE; unsigned int buf_index = (id / 2) % TSOH_PER_PAGE; header = (txq->tsoh_buffer[page_index].esm_base + buf_index * TSOH_STD_SIZE); dma_addr = (txq->tsoh_buffer[page_index].esm_addr + buf_index * TSOH_STD_SIZE); map = txq->tsoh_buffer[page_index].esm_map; KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); } else { struct sfxge_tx_mapping *stmp = &txq->stmp[id]; /* We cannot use bus_dmamem_alloc() as that may sleep */ header = malloc(tso->header_len, M_SFXGE, M_NOWAIT); if (__predict_false(!header)) return (ENOMEM); rc = bus_dmamap_load(txq->packet_dma_tag, stmp->map, header, tso->header_len, tso_map_long_header, &dma_addr, BUS_DMA_NOWAIT); if (__predict_false(dma_addr == 0)) { if (rc == 0) { /* Succeeded but got >1 segment */ bus_dmamap_unload(txq->packet_dma_tag, stmp->map); rc = EINVAL; } free(header, M_SFXGE); return (rc); } map = stmp->map; txq->tso_long_headers++; stmp->u.heap_buf = header; stmp->flags = TX_BUF_UNMAP; } tsoh_th = (struct tcphdr *)(header + tso->tcph_off); /* Copy and update the headers. */ m_copydata(tso->mbuf, 0, tso->header_len, header); tsoh_th->th_seq = htonl(tso->seqnum); tso->seqnum += tso->seg_size; if (tso->out_len > tso->seg_size) { /* This packet will not finish the TSO burst. */ ip_length = tso->header_len - tso->nh_off + tso->seg_size; tsoh_th->th_flags &= ~(TH_FIN | TH_PUSH); } else { /* This packet will be the last in the TSO burst. */ ip_length = tso->header_len - tso->nh_off + tso->out_len; } if (tso->protocol == htons(ETHERTYPE_IP)) { struct ip *tsoh_iph = (struct ip *)(header + tso->nh_off); tsoh_iph->ip_len = htons(ip_length); /* XXX We should increment ip_id, but FreeBSD doesn't * currently allocate extra IDs for multiple segments. */ } else { struct ip6_hdr *tsoh_iph = (struct ip6_hdr *)(header + tso->nh_off); tsoh_iph->ip6_plen = htons(ip_length - sizeof(*tsoh_iph)); } /* Make the header visible to the hardware. */ bus_dmamap_sync(txq->packet_dma_tag, map, BUS_DMASYNC_PREWRITE); /* Form a descriptor for this header. */ desc = &txq->pend_desc[txq->n_pend_desc++]; efx_tx_qdesc_dma_create(txq->common, dma_addr, tso->header_len, 0, desc); id = (id + 1) & txq->ptr_mask; + + tso->segs_space = UINT_MAX; } tso->packet_space = tso->seg_size; txq->tso_packets++; *idp = id; return (0); } static int sfxge_tx_queue_tso(struct sfxge_txq *txq, struct mbuf *mbuf, const bus_dma_segment_t *dma_seg, int n_dma_seg, int vlan_tagged) { struct sfxge_tso_state tso; unsigned int id; unsigned skipped = 0; tso_start(txq, &tso, dma_seg, mbuf); while (dma_seg->ds_len + skipped <= tso.header_len) { skipped += dma_seg->ds_len; --n_dma_seg; KASSERT(n_dma_seg, ("no payload found in TSO packet")); ++dma_seg; } tso.in_len = dma_seg->ds_len - (tso.header_len - skipped); tso.dma_addr = dma_seg->ds_addr + (tso.header_len - skipped); id = (txq->added + vlan_tagged) & txq->ptr_mask; if (__predict_false(tso_start_new_packet(txq, &tso, &id))) return (-1); while (1) { tso_fill_packet_with_fragment(txq, &tso); /* Exactly one DMA descriptor is added */ KASSERT(txq->stmp[id].flags == 0, ("stmp flags are not 0")); id = (id + 1) & txq->ptr_mask; /* Move onto the next fragment? */ if (tso.in_len == 0) { --n_dma_seg; if (n_dma_seg == 0) break; ++dma_seg; tso.in_len = dma_seg->ds_len; tso.dma_addr = dma_seg->ds_addr; } /* End of packet? */ - if (tso.packet_space == 0) { + if ((tso.packet_space == 0) | (tso.segs_space == 0)) { + unsigned int n_fatso_opt_desc = + (tso.fw_assisted & SFXGE_FATSOV2) ? + EFX_TX_FATSOV2_OPT_NDESCS : + (tso.fw_assisted & SFXGE_FATSOV1) ? 1 : 0; + /* If the queue is now full due to tiny MSS, * or we can't create another header, discard * the remainder of the input mbuf but do not * roll back the work we have done. */ - if (txq->n_pend_desc + tso.fw_assisted + - 1 /* header */ + n_dma_seg > - txq->max_pkt_desc) { + if (txq->n_pend_desc + n_fatso_opt_desc + + 1 /* header */ + n_dma_seg > txq->max_pkt_desc) { txq->tso_pdrop_too_many++; break; } if (__predict_false(tso_start_new_packet(txq, &tso, &id))) { txq->tso_pdrop_no_rsrc++; break; } } } txq->tso_bursts++; return (id); } static void sfxge_tx_qunblock(struct sfxge_txq *txq) { struct sfxge_softc *sc; struct sfxge_evq *evq; sc = txq->sc; evq = sc->evq[txq->evq_index]; SFXGE_EVQ_LOCK_ASSERT_OWNED(evq); if (__predict_false(txq->init_state != SFXGE_TXQ_STARTED)) return; SFXGE_TXQ_LOCK(txq); if (txq->blocked) { unsigned int level; level = txq->added - txq->completed; if (level <= SFXGE_TXQ_UNBLOCK_LEVEL(txq->entries)) { /* reaped must be in sync with blocked */ sfxge_tx_qreap(txq); txq->blocked = 0; } } sfxge_tx_qdpl_service(txq); /* note: lock has been dropped */ } void sfxge_tx_qflush_done(struct sfxge_txq *txq) { txq->flush_state = SFXGE_FLUSH_DONE; } static void sfxge_tx_qstop(struct sfxge_softc *sc, unsigned int index) { struct sfxge_txq *txq; struct sfxge_evq *evq; unsigned int count; SFXGE_ADAPTER_LOCK_ASSERT_OWNED(sc); txq = sc->txq[index]; evq = sc->evq[txq->evq_index]; SFXGE_EVQ_LOCK(evq); SFXGE_TXQ_LOCK(txq); KASSERT(txq->init_state == SFXGE_TXQ_STARTED, ("txq->init_state != SFXGE_TXQ_STARTED")); txq->init_state = SFXGE_TXQ_INITIALIZED; if (txq->flush_state != SFXGE_FLUSH_DONE) { txq->flush_state = SFXGE_FLUSH_PENDING; SFXGE_EVQ_UNLOCK(evq); SFXGE_TXQ_UNLOCK(txq); /* Flush the transmit queue. */ if (efx_tx_qflush(txq->common) != 0) { log(LOG_ERR, "%s: Flushing Tx queue %u failed\n", device_get_nameunit(sc->dev), index); txq->flush_state = SFXGE_FLUSH_DONE; } else { count = 0; do { /* Spin for 100ms. */ DELAY(100000); if (txq->flush_state != SFXGE_FLUSH_PENDING) break; } while (++count < 20); } SFXGE_EVQ_LOCK(evq); SFXGE_TXQ_LOCK(txq); KASSERT(txq->flush_state != SFXGE_FLUSH_FAILED, ("txq->flush_state == SFXGE_FLUSH_FAILED")); if (txq->flush_state != SFXGE_FLUSH_DONE) { /* Flush timeout */ log(LOG_ERR, "%s: Cannot flush Tx queue %u\n", device_get_nameunit(sc->dev), index); txq->flush_state = SFXGE_FLUSH_DONE; } } txq->blocked = 0; txq->pending = txq->added; sfxge_tx_qcomplete(txq, evq); KASSERT(txq->completed == txq->added, ("txq->completed != txq->added")); sfxge_tx_qreap(txq); KASSERT(txq->reaped == txq->completed, ("txq->reaped != txq->completed")); txq->added = 0; txq->pending = 0; txq->completed = 0; txq->reaped = 0; /* Destroy the common code transmit queue. */ efx_tx_qdestroy(txq->common); txq->common = NULL; efx_sram_buf_tbl_clear(sc->enp, txq->buf_base_id, EFX_TXQ_NBUFS(sc->txq_entries)); SFXGE_EVQ_UNLOCK(evq); SFXGE_TXQ_UNLOCK(txq); } +/* + * Estimate maximum number of Tx descriptors required for TSO packet. + * With minimum MSS and maximum mbuf length we might need more (even + * than a ring-ful of descriptors), but this should not happen in + * practice except due to deliberate attack. In that case we will + * truncate the output at a packet boundary. + */ +static unsigned int +sfxge_tx_max_pkt_desc(const struct sfxge_softc *sc, enum sfxge_txq_type type, + unsigned int tso_fw_assisted) +{ + /* One descriptor for every input fragment */ + unsigned int max_descs = SFXGE_TX_MAPPING_MAX_SEG; + unsigned int sw_tso_max_descs; + unsigned int fa_tso_v1_max_descs = 0; + unsigned int fa_tso_v2_max_descs = 0; + + /* VLAN tagging Tx option descriptor may be required */ + if (efx_nic_cfg_get(sc->enp)->enc_hw_tx_insert_vlan_enabled) + max_descs++; + + if (type == SFXGE_TXQ_IP_TCP_UDP_CKSUM) { + /* + * Plus header and payload descriptor for each output segment. + * Minus one since header fragment is already counted. + * Even if FATSO is used, we should be ready to fallback + * to do it in the driver. + */ + sw_tso_max_descs = SFXGE_TSO_MAX_SEGS * 2 - 1; + + /* FW assisted TSOv1 requires one more descriptor per segment + * in comparison to SW TSO */ + if (tso_fw_assisted & SFXGE_FATSOV1) + fa_tso_v1_max_descs = + sw_tso_max_descs + SFXGE_TSO_MAX_SEGS; + + /* FW assisted TSOv2 requires 3 (2 FATSO plus header) extra + * descriptors per superframe limited by number of DMA fetches + * per packet. The first packet header is already counted. + */ + if (tso_fw_assisted & SFXGE_FATSOV2) { + fa_tso_v2_max_descs = + howmany(SFXGE_TX_MAPPING_MAX_SEG, + EFX_TX_FATSOV2_DMA_SEGS_PER_PKT_MAX - 1) * + (EFX_TX_FATSOV2_OPT_NDESCS + 1) - 1; + } + + max_descs += MAX(sw_tso_max_descs, + MAX(fa_tso_v1_max_descs, fa_tso_v2_max_descs)); + } + + return (max_descs); +} + static int sfxge_tx_qstart(struct sfxge_softc *sc, unsigned int index) { struct sfxge_txq *txq; efsys_mem_t *esmp; uint16_t flags; + unsigned int tso_fw_assisted; struct sfxge_evq *evq; unsigned int desc_index; int rc; SFXGE_ADAPTER_LOCK_ASSERT_OWNED(sc); txq = sc->txq[index]; esmp = &txq->mem; evq = sc->evq[txq->evq_index]; KASSERT(txq->init_state == SFXGE_TXQ_INITIALIZED, ("txq->init_state != SFXGE_TXQ_INITIALIZED")); KASSERT(evq->init_state == SFXGE_EVQ_STARTED, ("evq->init_state != SFXGE_EVQ_STARTED")); /* Program the buffer table. */ if ((rc = efx_sram_buf_tbl_set(sc->enp, txq->buf_base_id, esmp, EFX_TXQ_NBUFS(sc->txq_entries))) != 0) return (rc); /* Determine the kind of queue we are creating. */ + tso_fw_assisted = 0; switch (txq->type) { case SFXGE_TXQ_NON_CKSUM: flags = 0; break; case SFXGE_TXQ_IP_CKSUM: flags = EFX_TXQ_CKSUM_IPV4; break; case SFXGE_TXQ_IP_TCP_UDP_CKSUM: flags = EFX_TXQ_CKSUM_IPV4 | EFX_TXQ_CKSUM_TCPUDP; + tso_fw_assisted = sc->tso_fw_assisted; + if (tso_fw_assisted & SFXGE_FATSOV2) + flags |= EFX_TXQ_FATSOV2; break; default: KASSERT(0, ("Impossible TX queue")); flags = 0; break; } /* Create the common code transmit queue. */ if ((rc = efx_tx_qcreate(sc->enp, index, txq->type, esmp, sc->txq_entries, txq->buf_base_id, flags, evq->common, - &txq->common, &desc_index)) != 0) - goto fail; + &txq->common, &desc_index)) != 0) { + /* Retry if no FATSOv2 resources, otherwise fail */ + if ((rc != ENOSPC) || (~flags & EFX_TXQ_FATSOV2)) + goto fail; + /* Looks like all FATSOv2 contexts are used */ + flags &= ~EFX_TXQ_FATSOV2; + tso_fw_assisted &= ~SFXGE_FATSOV2; + if ((rc = efx_tx_qcreate(sc->enp, index, txq->type, esmp, + sc->txq_entries, txq->buf_base_id, flags, evq->common, + &txq->common, &desc_index)) != 0) + goto fail; + } + /* Initialise queue descriptor indexes */ txq->added = txq->pending = txq->completed = txq->reaped = desc_index; SFXGE_TXQ_LOCK(txq); /* Enable the transmit queue. */ efx_tx_qenable(txq->common); txq->init_state = SFXGE_TXQ_STARTED; txq->flush_state = SFXGE_FLUSH_REQUIRED; + txq->tso_fw_assisted = tso_fw_assisted; + txq->max_pkt_desc = sfxge_tx_max_pkt_desc(sc, txq->type, + tso_fw_assisted); + SFXGE_TXQ_UNLOCK(txq); return (0); fail: efx_sram_buf_tbl_clear(sc->enp, txq->buf_base_id, EFX_TXQ_NBUFS(sc->txq_entries)); return (rc); } void sfxge_tx_stop(struct sfxge_softc *sc) { int index; index = sc->txq_count; while (--index >= 0) sfxge_tx_qstop(sc, index); /* Tear down the transmit module */ efx_tx_fini(sc->enp); } int sfxge_tx_start(struct sfxge_softc *sc) { int index; int rc; /* Initialize the common code transmit module. */ if ((rc = efx_tx_init(sc->enp)) != 0) return (rc); for (index = 0; index < sc->txq_count; index++) { if ((rc = sfxge_tx_qstart(sc, index)) != 0) goto fail; } return (0); fail: while (--index >= 0) sfxge_tx_qstop(sc, index); efx_tx_fini(sc->enp); return (rc); } static int sfxge_txq_stat_init(struct sfxge_txq *txq, struct sysctl_oid *txq_node) { struct sysctl_ctx_list *ctx = device_get_sysctl_ctx(txq->sc->dev); struct sysctl_oid *stat_node; unsigned int id; stat_node = SYSCTL_ADD_NODE(ctx, SYSCTL_CHILDREN(txq_node), OID_AUTO, "stats", CTLFLAG_RD, NULL, "Tx queue statistics"); if (stat_node == NULL) return (ENOMEM); for (id = 0; id < nitems(sfxge_tx_stats); id++) { SYSCTL_ADD_ULONG( ctx, SYSCTL_CHILDREN(stat_node), OID_AUTO, sfxge_tx_stats[id].name, CTLFLAG_RD | CTLFLAG_STATS, (unsigned long *)((caddr_t)txq + sfxge_tx_stats[id].offset), ""); } return (0); } /** * Destroy a transmit queue. */ static void sfxge_tx_qfini(struct sfxge_softc *sc, unsigned int index) { struct sfxge_txq *txq; unsigned int nmaps; txq = sc->txq[index]; KASSERT(txq->init_state == SFXGE_TXQ_INITIALIZED, ("txq->init_state != SFXGE_TXQ_INITIALIZED")); if (txq->type == SFXGE_TXQ_IP_TCP_UDP_CKSUM) tso_fini(txq); /* Free the context arrays. */ free(txq->pend_desc, M_SFXGE); nmaps = sc->txq_entries; while (nmaps-- != 0) bus_dmamap_destroy(txq->packet_dma_tag, txq->stmp[nmaps].map); free(txq->stmp, M_SFXGE); /* Release DMA memory mapping. */ sfxge_dma_free(&txq->mem); sc->txq[index] = NULL; SFXGE_TXQ_LOCK_DESTROY(txq); free(txq, M_SFXGE); } -/* - * Estimate maximum number of Tx descriptors required for TSO packet. - * With minimum MSS and maximum mbuf length we might need more (even - * than a ring-ful of descriptors), but this should not happen in - * practice except due to deliberate attack. In that case we will - * truncate the output at a packet boundary. - */ -static unsigned int -sfxge_tx_max_pkt_desc(const struct sfxge_softc *sc, enum sfxge_txq_type type) -{ - /* One descriptor for every input fragment */ - unsigned int max_descs = SFXGE_TX_MAPPING_MAX_SEG; - - /* VLAN tagging Tx option descriptor may be required */ - if (efx_nic_cfg_get(sc->enp)->enc_hw_tx_insert_vlan_enabled) - max_descs++; - - if (type == SFXGE_TXQ_IP_TCP_UDP_CKSUM) { - /* - * Plus header and payload descriptor for each output segment. - * Minus one since header fragment is already counted. - */ - max_descs += SFXGE_TSO_MAX_SEGS * 2 - 1; - - /* FW assisted TSO requires one more descriptor per segment */ - if (sc->tso_fw_assisted) - max_descs += SFXGE_TSO_MAX_SEGS; - } - - return (max_descs); -} - static int sfxge_tx_qinit(struct sfxge_softc *sc, unsigned int txq_index, enum sfxge_txq_type type, unsigned int evq_index) { char name[16]; struct sysctl_ctx_list *ctx = device_get_sysctl_ctx(sc->dev); struct sysctl_oid *txq_node; struct sfxge_txq *txq; struct sfxge_evq *evq; struct sfxge_tx_dpl *stdp; struct sysctl_oid *dpl_node; efsys_mem_t *esmp; unsigned int nmaps; int rc; txq = malloc(sizeof(struct sfxge_txq), M_SFXGE, M_ZERO | M_WAITOK); txq->sc = sc; txq->entries = sc->txq_entries; txq->ptr_mask = txq->entries - 1; sc->txq[txq_index] = txq; esmp = &txq->mem; evq = sc->evq[evq_index]; /* Allocate and zero DMA space for the descriptor ring. */ if ((rc = sfxge_dma_alloc(sc, EFX_TXQ_SIZE(sc->txq_entries), esmp)) != 0) return (rc); /* Allocate buffer table entries. */ sfxge_sram_buf_tbl_alloc(sc, EFX_TXQ_NBUFS(sc->txq_entries), &txq->buf_base_id); /* Create a DMA tag for packet mappings. */ if (bus_dma_tag_create(sc->parent_dma_tag, 1, 0x1000, MIN(0x3FFFFFFFFFFFUL, BUS_SPACE_MAXADDR), BUS_SPACE_MAXADDR, NULL, NULL, 0x11000, SFXGE_TX_MAPPING_MAX_SEG, 0x1000, 0, NULL, NULL, &txq->packet_dma_tag) != 0) { device_printf(sc->dev, "Couldn't allocate txq DMA tag\n"); rc = ENOMEM; goto fail; } /* Allocate pending descriptor array for batching writes. */ txq->pend_desc = malloc(sizeof(efx_desc_t) * sc->txq_entries, M_SFXGE, M_ZERO | M_WAITOK); /* Allocate and initialise mbuf DMA mapping array. */ txq->stmp = malloc(sizeof(struct sfxge_tx_mapping) * sc->txq_entries, M_SFXGE, M_ZERO | M_WAITOK); for (nmaps = 0; nmaps < sc->txq_entries; nmaps++) { rc = bus_dmamap_create(txq->packet_dma_tag, 0, &txq->stmp[nmaps].map); if (rc != 0) goto fail2; } snprintf(name, sizeof(name), "%u", txq_index); txq_node = SYSCTL_ADD_NODE(ctx, SYSCTL_CHILDREN(sc->txqs_node), OID_AUTO, name, CTLFLAG_RD, NULL, ""); if (txq_node == NULL) { rc = ENOMEM; goto fail_txq_node; } if (type == SFXGE_TXQ_IP_TCP_UDP_CKSUM && (rc = tso_init(txq)) != 0) goto fail3; if (sfxge_tx_dpl_get_max <= 0) { log(LOG_ERR, "%s=%d must be greater than 0", SFXGE_PARAM_TX_DPL_GET_MAX, sfxge_tx_dpl_get_max); rc = EINVAL; goto fail_tx_dpl_get_max; } if (sfxge_tx_dpl_get_non_tcp_max <= 0) { log(LOG_ERR, "%s=%d must be greater than 0", SFXGE_PARAM_TX_DPL_GET_NON_TCP_MAX, sfxge_tx_dpl_get_non_tcp_max); rc = EINVAL; goto fail_tx_dpl_get_max; } if (sfxge_tx_dpl_put_max < 0) { log(LOG_ERR, "%s=%d must be greater or equal to 0", SFXGE_PARAM_TX_DPL_PUT_MAX, sfxge_tx_dpl_put_max); rc = EINVAL; goto fail_tx_dpl_put_max; } /* Initialize the deferred packet list. */ stdp = &txq->dpl; stdp->std_put_max = sfxge_tx_dpl_put_max; stdp->std_get_max = sfxge_tx_dpl_get_max; stdp->std_get_non_tcp_max = sfxge_tx_dpl_get_non_tcp_max; stdp->std_getp = &stdp->std_get; SFXGE_TXQ_LOCK_INIT(txq, device_get_nameunit(sc->dev), txq_index); dpl_node = SYSCTL_ADD_NODE(ctx, SYSCTL_CHILDREN(txq_node), OID_AUTO, "dpl", CTLFLAG_RD, NULL, "Deferred packet list statistics"); if (dpl_node == NULL) { rc = ENOMEM; goto fail_dpl_node; } SYSCTL_ADD_UINT(ctx, SYSCTL_CHILDREN(dpl_node), OID_AUTO, "get_count", CTLFLAG_RD | CTLFLAG_STATS, &stdp->std_get_count, 0, ""); SYSCTL_ADD_UINT(ctx, SYSCTL_CHILDREN(dpl_node), OID_AUTO, "get_non_tcp_count", CTLFLAG_RD | CTLFLAG_STATS, &stdp->std_get_non_tcp_count, 0, ""); SYSCTL_ADD_UINT(ctx, SYSCTL_CHILDREN(dpl_node), OID_AUTO, "get_hiwat", CTLFLAG_RD | CTLFLAG_STATS, &stdp->std_get_hiwat, 0, ""); SYSCTL_ADD_UINT(ctx, SYSCTL_CHILDREN(dpl_node), OID_AUTO, "put_hiwat", CTLFLAG_RD | CTLFLAG_STATS, &stdp->std_put_hiwat, 0, ""); rc = sfxge_txq_stat_init(txq, txq_node); if (rc != 0) goto fail_txq_stat_init; txq->type = type; txq->evq_index = evq_index; txq->txq_index = txq_index; txq->init_state = SFXGE_TXQ_INITIALIZED; txq->hw_vlan_tci = 0; - txq->max_pkt_desc = sfxge_tx_max_pkt_desc(sc, type); - return (0); fail_txq_stat_init: fail_dpl_node: fail_tx_dpl_put_max: fail_tx_dpl_get_max: fail3: fail_txq_node: free(txq->pend_desc, M_SFXGE); fail2: while (nmaps-- != 0) bus_dmamap_destroy(txq->packet_dma_tag, txq->stmp[nmaps].map); free(txq->stmp, M_SFXGE); bus_dma_tag_destroy(txq->packet_dma_tag); fail: sfxge_dma_free(esmp); return (rc); } static int sfxge_tx_stat_handler(SYSCTL_HANDLER_ARGS) { struct sfxge_softc *sc = arg1; unsigned int id = arg2; unsigned long sum; unsigned int index; /* Sum across all TX queues */ sum = 0; for (index = 0; index < sc->txq_count; index++) sum += *(unsigned long *)((caddr_t)sc->txq[index] + sfxge_tx_stats[id].offset); return (SYSCTL_OUT(req, &sum, sizeof(sum))); } static void sfxge_tx_stat_init(struct sfxge_softc *sc) { struct sysctl_ctx_list *ctx = device_get_sysctl_ctx(sc->dev); struct sysctl_oid_list *stat_list; unsigned int id; stat_list = SYSCTL_CHILDREN(sc->stats_node); for (id = 0; id < nitems(sfxge_tx_stats); id++) { SYSCTL_ADD_PROC( ctx, stat_list, OID_AUTO, sfxge_tx_stats[id].name, CTLTYPE_ULONG|CTLFLAG_RD, sc, id, sfxge_tx_stat_handler, "LU", ""); } } uint64_t sfxge_tx_get_drops(struct sfxge_softc *sc) { unsigned int index; uint64_t drops = 0; struct sfxge_txq *txq; /* Sum across all TX queues */ for (index = 0; index < sc->txq_count; index++) { txq = sc->txq[index]; /* * In theory, txq->put_overflow and txq->netdown_drops * should use atomic operation and other should be * obtained under txq lock, but it is just statistics. */ drops += txq->drops + txq->get_overflow + txq->get_non_tcp_overflow + txq->put_overflow + txq->netdown_drops + txq->tso_pdrop_too_many + txq->tso_pdrop_no_rsrc; } return (drops); } void sfxge_tx_fini(struct sfxge_softc *sc) { int index; index = sc->txq_count; while (--index >= 0) sfxge_tx_qfini(sc, index); sc->txq_count = 0; } int sfxge_tx_init(struct sfxge_softc *sc) { const efx_nic_cfg_t *encp = efx_nic_cfg_get(sc->enp); struct sfxge_intr *intr; int index; int rc; intr = &sc->intr; KASSERT(intr->state == SFXGE_INTR_INITIALIZED, ("intr->state != SFXGE_INTR_INITIALIZED")); sc->txq_count = SFXGE_TXQ_NTYPES - 1 + sc->intr.n_alloc; sc->tso_fw_assisted = sfxge_tso_fw_assisted; - if (sc->tso_fw_assisted) - sc->tso_fw_assisted = - (encp->enc_features & EFX_FEATURE_FW_ASSISTED_TSO) && - (encp->enc_fw_assisted_tso_enabled); + if ((~encp->enc_features & EFX_FEATURE_FW_ASSISTED_TSO) || + (!encp->enc_fw_assisted_tso_enabled)) + sc->tso_fw_assisted &= ~SFXGE_FATSOV1; + if ((~encp->enc_features & EFX_FEATURE_FW_ASSISTED_TSO_V2) || + (!encp->enc_fw_assisted_tso_v2_enabled)) + sc->tso_fw_assisted &= ~SFXGE_FATSOV2; sc->txqs_node = SYSCTL_ADD_NODE( device_get_sysctl_ctx(sc->dev), SYSCTL_CHILDREN(device_get_sysctl_tree(sc->dev)), OID_AUTO, "txq", CTLFLAG_RD, NULL, "Tx queues"); if (sc->txqs_node == NULL) { rc = ENOMEM; goto fail_txq_node; } /* Initialize the transmit queues */ if ((rc = sfxge_tx_qinit(sc, SFXGE_TXQ_NON_CKSUM, SFXGE_TXQ_NON_CKSUM, 0)) != 0) goto fail; if ((rc = sfxge_tx_qinit(sc, SFXGE_TXQ_IP_CKSUM, SFXGE_TXQ_IP_CKSUM, 0)) != 0) goto fail2; for (index = 0; index < sc->txq_count - SFXGE_TXQ_NTYPES + 1; index++) { if ((rc = sfxge_tx_qinit(sc, SFXGE_TXQ_NTYPES - 1 + index, SFXGE_TXQ_IP_TCP_UDP_CKSUM, index)) != 0) goto fail3; } sfxge_tx_stat_init(sc); return (0); fail3: while (--index >= 0) sfxge_tx_qfini(sc, SFXGE_TXQ_IP_TCP_UDP_CKSUM + index); sfxge_tx_qfini(sc, SFXGE_TXQ_IP_CKSUM); fail2: sfxge_tx_qfini(sc, SFXGE_TXQ_NON_CKSUM); fail: fail_txq_node: sc->txq_count = 0; return (rc); } Index: head/sys/dev/sfxge/sfxge_tx.h =================================================================== --- head/sys/dev/sfxge/sfxge_tx.h (revision 294076) +++ head/sys/dev/sfxge/sfxge_tx.h (revision 294077) @@ -1,243 +1,244 @@ /*- * Copyright (c) 2010-2015 Solarflare Communications Inc. * All rights reserved. * * This software was developed in part by Philip Paeps under contract for * Solarflare Communications, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The views and conclusions contained in the software and documentation are * those of the authors and should not be interpreted as representing official * policies, either expressed or implied, of the FreeBSD Project. * * $FreeBSD$ */ #ifndef _SFXGE_TX_H #define _SFXGE_TX_H #include #include #include /* If defined, parse TX packets directly in if_transmit * for better cache locality and reduced time under TX lock */ #define SFXGE_TX_PARSE_EARLY 1 /* Maximum size of TSO packet */ #define SFXGE_TSO_MAX_SIZE (65535) /* * Maximum number of segments to be created for a TSO packet. * Allow for a reasonable minimum MSS of 512. */ #define SFXGE_TSO_MAX_SEGS howmany(SFXGE_TSO_MAX_SIZE, 512) /* Maximum number of DMA segments needed to map an mbuf chain. With * TSO, the mbuf length may be just over 64K, divided into 2K mbuf * clusters taking into account that the first may be not 2K cluster * boundary aligned. * Packet header may be split into two segments because of, for example, * VLAN header insertion. * The chain could be longer than this initially, but can be shortened * with m_collapse(). */ #define SFXGE_TX_MAPPING_MAX_SEG \ (2 + howmany(SFXGE_TSO_MAX_SIZE, MCLBYTES) + 1) /* * Buffer mapping flags. * * Buffers and DMA mappings must be freed when the last descriptor * referring to them is completed. Set the TX_BUF_UNMAP and * TX_BUF_MBUF flags on the last descriptor generated for an mbuf * chain. Set only the TX_BUF_UNMAP flag on a descriptor referring to * a heap buffer. */ enum sfxge_tx_buf_flags { TX_BUF_UNMAP = 1, TX_BUF_MBUF = 2, }; /* * Buffer mapping information for descriptors in flight. */ struct sfxge_tx_mapping { union { struct mbuf *mbuf; caddr_t heap_buf; } u; bus_dmamap_t map; enum sfxge_tx_buf_flags flags; }; #define SFXGE_TX_DPL_GET_PKT_LIMIT_DEFAULT (64 * 1024) #define SFXGE_TX_DPL_GET_NON_TCP_PKT_LIMIT_DEFAULT 1024 #define SFXGE_TX_DPL_PUT_PKT_LIMIT_DEFAULT 1024 /* * Deferred packet list. */ struct sfxge_tx_dpl { unsigned int std_get_max; /* Maximum number of packets * in get list */ unsigned int std_get_non_tcp_max; /* Maximum number * of non-TCP packets * in get list */ unsigned int std_put_max; /* Maximum number of packets * in put list */ uintptr_t std_put; /* Head of put list. */ struct mbuf *std_get; /* Head of get list. */ struct mbuf **std_getp; /* Tail of get list. */ unsigned int std_get_count; /* Packets in get list. */ unsigned int std_get_non_tcp_count; /* Non-TCP packets * in get list */ unsigned int std_get_hiwat; /* Packets in get list * high watermark */ unsigned int std_put_hiwat; /* Packets in put list * high watermark */ }; #define SFXGE_TX_BUFFER_SIZE 0x400 #define SFXGE_TX_HEADER_SIZE 0x100 #define SFXGE_TX_COPY_THRESHOLD 0x200 enum sfxge_txq_state { SFXGE_TXQ_UNINITIALIZED = 0, SFXGE_TXQ_INITIALIZED, SFXGE_TXQ_STARTED }; enum sfxge_txq_type { SFXGE_TXQ_NON_CKSUM = 0, SFXGE_TXQ_IP_CKSUM, SFXGE_TXQ_IP_TCP_UDP_CKSUM, SFXGE_TXQ_NTYPES }; #define SFXGE_TXQ_UNBLOCK_LEVEL(_entries) (EFX_TXQ_LIMIT(_entries) / 4) #define SFXGE_TX_BATCH 64 #define SFXGE_TXQ_LOCK_INIT(_txq, _ifname, _txq_index) \ do { \ struct sfxge_txq *__txq = (_txq); \ \ snprintf((__txq)->lock_name, \ sizeof((__txq)->lock_name), \ "%s:txq%u", (_ifname), (_txq_index)); \ mtx_init(&(__txq)->lock, (__txq)->lock_name, \ NULL, MTX_DEF); \ } while (B_FALSE) #define SFXGE_TXQ_LOCK_DESTROY(_txq) \ mtx_destroy(&(_txq)->lock) #define SFXGE_TXQ_LOCK(_txq) \ mtx_lock(&(_txq)->lock) #define SFXGE_TXQ_TRYLOCK(_txq) \ mtx_trylock(&(_txq)->lock) #define SFXGE_TXQ_UNLOCK(_txq) \ mtx_unlock(&(_txq)->lock) #define SFXGE_TXQ_LOCK_ASSERT_OWNED(_txq) \ mtx_assert(&(_txq)->lock, MA_OWNED) #define SFXGE_TXQ_LOCK_ASSERT_NOTOWNED(_txq) \ mtx_assert(&(_txq)->lock, MA_NOTOWNED) struct sfxge_txq { /* The following fields should be written very rarely */ struct sfxge_softc *sc; enum sfxge_txq_state init_state; enum sfxge_flush_state flush_state; + unsigned int tso_fw_assisted; enum sfxge_txq_type type; unsigned int txq_index; unsigned int evq_index; efsys_mem_t mem; unsigned int buf_base_id; unsigned int entries; unsigned int ptr_mask; unsigned int max_pkt_desc; struct sfxge_tx_mapping *stmp; /* Packets in flight. */ bus_dma_tag_t packet_dma_tag; efx_desc_t *pend_desc; efx_txq_t *common; efsys_mem_t *tsoh_buffer; char lock_name[SFXGE_LOCK_NAME_MAX]; /* This field changes more often and is read regularly on both * the initiation and completion paths */ int blocked __aligned(CACHE_LINE_SIZE); /* The following fields change more often, and are used mostly * on the initiation path */ struct mtx lock __aligned(CACHE_LINE_SIZE); struct sfxge_tx_dpl dpl; /* Deferred packet list. */ unsigned int n_pend_desc; unsigned int added; unsigned int reaped; /* The last VLAN TCI seen on the queue if FW-assisted tagging is used */ uint16_t hw_vlan_tci; /* Statistics */ unsigned long tso_bursts; unsigned long tso_packets; unsigned long tso_long_headers; unsigned long collapses; unsigned long drops; unsigned long get_overflow; unsigned long get_non_tcp_overflow; unsigned long put_overflow; unsigned long netdown_drops; unsigned long tso_pdrop_too_many; unsigned long tso_pdrop_no_rsrc; /* The following fields change more often, and are used mostly * on the completion path */ unsigned int pending __aligned(CACHE_LINE_SIZE); unsigned int completed; struct sfxge_txq *next; }; struct sfxge_evq; extern uint64_t sfxge_tx_get_drops(struct sfxge_softc *sc); extern int sfxge_tx_init(struct sfxge_softc *sc); extern void sfxge_tx_fini(struct sfxge_softc *sc); extern int sfxge_tx_start(struct sfxge_softc *sc); extern void sfxge_tx_stop(struct sfxge_softc *sc); extern void sfxge_tx_qcomplete(struct sfxge_txq *txq, struct sfxge_evq *evq); extern void sfxge_tx_qflush_done(struct sfxge_txq *txq); extern void sfxge_if_qflush(struct ifnet *ifp); extern int sfxge_if_transmit(struct ifnet *ifp, struct mbuf *m); #endif