Page MenuHomeFreeBSD
Differential D11378 Diff 31047 sys/dev/ixl/iwarp/README

Changeset View

Standalone View

View Options

sys/dev/ixl/iwarp/README

  • This file was added.
ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722
================================================================
July 20, 2017
Contents
========
- Prerequisites
- Building and Installation
- Testing
- Configuration
- Interoperability
- Known Issues
================================================================================
Prerequisites
-------------
- FreeBSD version 10.3 or 11.0, or later.
- Kernel configuration:
Please add the following kernel configuration options:
include GENERIC
options OFED
options OFED_DEBUG_INIT
options COMPAT_LINUXKPI
options SDP
options IPOIB_CM
options IXL_IW
nodevice ixl
nodevice ixlv
Note: IXL_IW is required for FreeBSD-CURRENT branch.
- For the iw_ixl driver to work, an if_ixl driver with iwarp interface
is required. The interface is available in if_ixl version 1.7.12 or later.
It should be enabled prior to usage, as the setting is switched off by
default. To enable iwarp compatibility, add
hw.ixl.enable_iwarp=1
to
/boot/loader.conf
The lan driver can be downloaded from
https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD
Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network
Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in
the ixl driver version 1.7.12-k or later.
There are some known issues with the interface on if_ixl-1.7.12. Please
use version 1.7.13 or later.
- fastreg memory mode in krping needs a patch applied to krping.
Refer to the 'Testing' and 'Known Issues' sections for details.
- fr test in krping in FreeBSD 10.3 needs a patch applied to krping.
Refer to 'Testing' and 'Known Issues' section for details.
Building and Installation
-------------------------
1. Untar ixl-<version>.tar.gz and iw_ixl-<version>.tar.gz
tar -xf ixl-<version>.tar.gz
tar -xf iw_ixl-<version>.tar.gz
2. Install the if_ixl driver:
cd ixl-<version>/src directory
make
make install
3. Install the iw_ixl driver:
cd iw_ixl-<version>/src
make clean
make IXL_DIR=$PATH_TO_IXL/ixl-<version>/src
make install
4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file
to the directory where manual pages are held on your system, for instance:
cp iw_ixl-<version>/doc/iw_ixl.4.gz /usr/share/man/man4/
For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow
the instruction from point 3 but ensure the correct path to if_ixl source
folder is supplied, for instance:
IXL_DIR=/usr/src/sys/dev/ixl/
Testing
-------
1. To load iw_ixl driver call:
kldload iw_ixl
If if_ixl is not already loaded, the system will load it on its own.
Please remember to add
hw.ixl.iwarp_enable=1
to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl
driver has the iwarp interface enabled.
2. To validate the load of the driver check:
sysctl -a | grep infiniband
a number of sys.class.infiniband should appear provided at least one
port of the X722 is up.
3. The source code for krping software is provided with the kernel in
/usr/src/sys/contrib/rdma/krping/. To compile the software, change directory
to /usr/src/sys/modules/rdma/krping/ and invoke:
make clean
make
make install
Note: In FreeBSD 10.3, some changes need to be applied to the krping.c file
in order to enable fastreg memory mode and/or fr test.
For details please refer to Known Issues section.
4. Start krping server on one machine:
echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping
5. Connect client from another machine:
echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping
================================================================================
Configuration
-------------
The following sysctl options are visible:
- hw.iw_ixl.max_ceq
determines the maximum number of msix vectors available to the driver
for CEQ usage.
- hw.iw_ixl.debug
defines level of debug messages.
- hw.iw_ixl.mpa_version
shows the current MPA version used.
The max_ceq setting may be changed by adding:
hw.iw_ixl.max_ceq=$value
to /boot/loader.conf file. The final number of CEQ is evaluated depending
on the available msix vectors, number of cpu cores, and hardware limits.
If max_ceq=0, the value is ignored.
The debug setting may be changed either by adding:
hw.iw_ixl.debug=$value
to the /boot/loader.conf file or by calling
sysctl hw.iw_ixl.debug=$value
The mpa_version may be changed by adding:
hw.iw_ixl.mpa_version=$value
to the /boot/loader.conf file.
Interoperability
----------------
To interoperate with Chelsio iWARP devices:
1. Load the ixl driver with parameter mpa_version set to 1.
Add the line:
hw.iw_ixl.mpa_version=1
to /boot/loader.conf
2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameters
c4iw_max_read_depth set to 63 and dack_mode set to 0.
================================================================================
Known Issues
------------
- Loopback is not supported.
- MTU changes are not supported.
- IPv6 is not supported.
- MW memory mode is not supported.
- MR memory mode supports only single buffer.
- The function ib_cq_resize is not supported.
- The max number of registered cq, qp, pd or mr reported by the device may
differ from the actual number of registrations achievable.
- A kernel crash may occur when trying to run krping without ensuring that the
two machines are able to ping each other.
- A kernel crash may occur when trying to load the iw_ixl driver when
hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13).
- A kernel crash may occur when loading the iw_ixl driver on a card that is
supported by if_ixl driver, but does not have iWARP capability (fixed with
if_ixl 1.7.13).
- On FreeBSD 10.3 you can run krping traffic with a Chelsio T5 card if
mpa_version on the X722 card is set to '1'. To set the MPA version, add:
hw.iw_ixl.mpa_version=1
to /boot/loader.conf.
Setting MPA version on both the X722 and the Chelsio T5 card to '2' may
result in a kernel crash.
On FreeBSD 11.0, krping traffic is possible with the default settings.
- Krping with fastreg memory mode will not work unless some changes are made
to krping. To workaround the issue, modify the krping_rdma_rkey function
such that, in the case of FASTREG memory mode, the ib_post_send function with
&cd->invalidate_wr parameter is not called during the first run of the function
- In FreeBSD 10.3, krping fr test is incorrectly implemented. Please consult
the krping.c file provided with the FreeBSD 11.0 kernel for the required
fixes. Needed changes include, but are not limited to:
- The fr test requires the server side (unlike what is written in
the comments).
function: krping_doit
function: krping_fr_test_server
function: krping_run_server
- The fr test require MR memory mode.
function: krping_doit
- Wrong device is queried on the client side when checking fastreg support.
function: fastreg_supported
- Note: the fr itself is designed to work endlessly, however lack of
wait_event_interruptible_timeout function support causes the test to do
nothing after the first iteration.
function: krping_fr_test
-------------------
Changes since 0.1.2
-------------------
- Fix for kernel crash on FreeBSD 11.0 when setting ixl3 down.
- Fix for krping test with fastreg and server_inv options enabled.
--------------------
Changes since 0.0.24
--------------------
- Added patch for 1.7.12 to fix a number of bugs (These changes are available
in the ixl driver, version 1.7.13 and later):
o kernel crash on loading iw_ixl on XL710 cards.
o kernel crash on loading iw_ixl with enable_iwarp tunable set to 0.
- MSIX vector utilization has been reworked.
- CPU affinity of the CEQ has been implemented.
- Added sysctl controls.
--------------------
Changes since 0.0.18
--------------------
- No patch for OFED is necessary for FreeBSD 11.0.
--------------------
Changes since 0.0.11
--------------------
- Support for FreeBSD 11.0 connection is added.
- No patch for OFED is necessary for FreeBSD 10.3.
================================================================================
Support
-------
For general information, go to the Intel support website at:
www.intel.com/support/
or the Intel Wired Networking project hosted by Sourceforge at:
http://sourceforge.net/projects/e1000
If an issue is identified with the released source code on a supported
kernel with a supported adapter, email the specific information related to the
issue to e1000-rdma@lists.sourceforge.net
================================================================================
License
-------
This software is available to you under a choice of one of two
licenses. You may choose to be licensed under the terms of the GNU
General Public License (GPL) Version 2, available from the file
COPYING in the main directory of this source tree, or the
OpenFabrics.org BSD license below:
Redistribution and use in source and binary forms, with or
without modification, are permitted provided that the following
conditions are met:
- Redistributions of source code must retain the above
copyright notice, this list of conditions and the following
disclaimer.
- Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials
provided with the distribution.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================================================
Trademarks
----------
Intel and Itanium are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and/or other countries.
* Other names and brands may be claimed as the property of others.