Changeset View
Changeset View
Standalone View
Standalone View
sys/dev/ixl/iwarp/README
- This file was added.
ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722 | |||||
================================================================ | |||||
July 20, 2017 | |||||
Contents | |||||
======== | |||||
- Prerequisites | |||||
- Building and Installation | |||||
- Testing | |||||
- Configuration | |||||
- Interoperability | |||||
- Known Issues | |||||
================================================================================ | |||||
Prerequisites | |||||
------------- | |||||
- FreeBSD version 10.3 or 11.0, or later. | |||||
- Kernel configuration: | |||||
Please add the following kernel configuration options: | |||||
include GENERIC | |||||
options OFED | |||||
options OFED_DEBUG_INIT | |||||
options COMPAT_LINUXKPI | |||||
options SDP | |||||
options IPOIB_CM | |||||
options IXL_IW | |||||
nodevice ixl | |||||
nodevice ixlv | |||||
Note: IXL_IW is required for FreeBSD-CURRENT branch. | |||||
- For the iw_ixl driver to work, an if_ixl driver with iwarp interface | |||||
is required. The interface is available in if_ixl version 1.7.12 or later. | |||||
It should be enabled prior to usage, as the setting is switched off by | |||||
default. To enable iwarp compatibility, add | |||||
hw.ixl.enable_iwarp=1 | |||||
to | |||||
/boot/loader.conf | |||||
The lan driver can be downloaded from | |||||
https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD | |||||
Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network | |||||
Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in | |||||
the ixl driver version 1.7.12-k or later. | |||||
There are some known issues with the interface on if_ixl-1.7.12. Please | |||||
use version 1.7.13 or later. | |||||
- fastreg memory mode in krping needs a patch applied to krping. | |||||
Refer to the 'Testing' and 'Known Issues' sections for details. | |||||
- fr test in krping in FreeBSD 10.3 needs a patch applied to krping. | |||||
Refer to 'Testing' and 'Known Issues' section for details. | |||||
Building and Installation | |||||
------------------------- | |||||
1. Untar ixl-<version>.tar.gz and iw_ixl-<version>.tar.gz | |||||
tar -xf ixl-<version>.tar.gz | |||||
tar -xf iw_ixl-<version>.tar.gz | |||||
2. Install the if_ixl driver: | |||||
cd ixl-<version>/src directory | |||||
make | |||||
make install | |||||
3. Install the iw_ixl driver: | |||||
cd iw_ixl-<version>/src | |||||
make clean | |||||
make IXL_DIR=$PATH_TO_IXL/ixl-<version>/src | |||||
make install | |||||
4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file | |||||
to the directory where manual pages are held on your system, for instance: | |||||
cp iw_ixl-<version>/doc/iw_ixl.4.gz /usr/share/man/man4/ | |||||
For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow | |||||
the instruction from point 3 but ensure the correct path to if_ixl source | |||||
folder is supplied, for instance: | |||||
IXL_DIR=/usr/src/sys/dev/ixl/ | |||||
Testing | |||||
------- | |||||
1. To load iw_ixl driver call: | |||||
kldload iw_ixl | |||||
If if_ixl is not already loaded, the system will load it on its own. | |||||
Please remember to add | |||||
hw.ixl.enable_iwarp=1 | |||||
to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl | |||||
driver has the iwarp interface enabled. | |||||
2. To validate the load of the driver check: | |||||
sysctl -a | grep infiniband | |||||
a number of sys.class.infiniband should appear provided at least one | |||||
port of the X722 is up. | |||||
3. The source code for krping software is provided with the kernel in | |||||
/usr/src/sys/contrib/rdma/krping/. To compile the software, change directory | |||||
to /usr/src/sys/modules/rdma/krping/ and invoke: | |||||
make clean | |||||
make | |||||
make install | |||||
Note: In FreeBSD 10.3, some changes need to be applied to the krping.c file | |||||
in order to enable fastreg memory mode and/or fr test. | |||||
For details please refer to Known Issues section. | |||||
4. Start krping server on one machine: | |||||
echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping | |||||
5. Connect client from another machine: | |||||
echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping | |||||
================================================================================ | |||||
Configuration | |||||
------------- | |||||
The following sysctl options are visible: | |||||
- hw.iw_ixl.max_ceq | |||||
determines the maximum number of msix vectors available to the driver | |||||
for CEQ usage. | |||||
- hw.iw_ixl.debug | |||||
defines level of debug messages. | |||||
- hw.iw_ixl.mpa_version | |||||
shows the current MPA version used. | |||||
The max_ceq setting may be changed by adding: | |||||
hw.iw_ixl.max_ceq=$value | |||||
to /boot/loader.conf file. The final number of CEQ is evaluated depending | |||||
on the available msix vectors, number of cpu cores, and hardware limits. | |||||
If max_ceq=0, the value is ignored. | |||||
The debug setting may be changed either by adding: | |||||
hw.iw_ixl.debug=$value | |||||
to the /boot/loader.conf file or by calling | |||||
sysctl hw.iw_ixl.debug=$value | |||||
The mpa_version may be changed by adding: | |||||
hw.iw_ixl.mpa_version=$value | |||||
to the /boot/loader.conf file. | |||||
Interoperability | |||||
---------------- | |||||
To interoperate with Chelsio iWARP devices: | |||||
1. Load the ixl driver with parameter mpa_version set to 1. | |||||
Add the line: | |||||
hw.iw_ixl.mpa_version=1 | |||||
to /boot/loader.conf | |||||
2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameters | |||||
c4iw_max_read_depth set to 63 and dack_mode set to 0. | |||||
================================================================================ | |||||
Known Issues | |||||
------------ | |||||
- Loopback is not supported. | |||||
- MTU changes are not supported. | |||||
- IPv6 is not supported. | |||||
- MW memory mode is not supported. | |||||
- MR memory mode supports only single buffer. | |||||
- The function ib_cq_resize is not supported. | |||||
- The max number of registered cq, qp, pd or mr reported by the device may | |||||
differ from the actual number of registrations achievable. | |||||
- A kernel crash may occur when trying to run krping without ensuring that the | |||||
two machines are able to ping each other. | |||||
- A kernel crash may occur when trying to load the iw_ixl driver when | |||||
hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13). | |||||
- A kernel crash may occur when loading the iw_ixl driver on a card that is | |||||
supported by if_ixl driver, but does not have iWARP capability (fixed with | |||||
if_ixl 1.7.13). | |||||
- On FreeBSD 10.3 you can run krping traffic with a Chelsio T5 card if | |||||
mpa_version on the X722 card is set to '1'. To set the MPA version, add: | |||||
hw.iw_ixl.mpa_version=1 | |||||
to /boot/loader.conf. | |||||
Setting MPA version on both the X722 and the Chelsio T5 card to '2' may | |||||
result in a kernel crash. | |||||
On FreeBSD 11.0, krping traffic is possible with the default settings. | |||||
- Krping with fastreg memory mode will not work unless some changes are made | |||||
to krping. To workaround the issue, modify the krping_rdma_rkey function | |||||
such that, in the case of FASTREG memory mode, the ib_post_send function with | |||||
&cd->invalidate_wr parameter is not called during the first run of the function | |||||
- In FreeBSD 10.3, krping fr test is incorrectly implemented. Please consult | |||||
the krping.c file provided with the FreeBSD 11.0 kernel for the required | |||||
fixes. Needed changes include, but are not limited to: | |||||
- The fr test requires the server side (unlike what is written in | |||||
the comments). | |||||
function: krping_doit | |||||
function: krping_fr_test_server | |||||
function: krping_run_server | |||||
- The fr test require MR memory mode. | |||||
function: krping_doit | |||||
- Wrong device is queried on the client side when checking fastreg support. | |||||
function: fastreg_supported | |||||
- Note: the fr itself is designed to work endlessly, however lack of | |||||
wait_event_interruptible_timeout function support causes the test to do | |||||
nothing after the first iteration. | |||||
function: krping_fr_test | |||||
------------------- | |||||
Changes since 0.1.2 | |||||
------------------- | |||||
- Fix for kernel crash on FreeBSD 11.0 when setting ixl3 down. | |||||
- Fix for krping test with fastreg and server_inv options enabled. | |||||
-------------------- | |||||
Changes since 0.0.24 | |||||
-------------------- | |||||
- Added patch for 1.7.12 to fix a number of bugs (These changes are available | |||||
in the ixl driver, version 1.7.13 and later): | |||||
o kernel crash on loading iw_ixl on XL710 cards. | |||||
o kernel crash on loading iw_ixl with enable_iwarp tunable set to 0. | |||||
- MSIX vector utilization has been reworked. | |||||
- CPU affinity of the CEQ has been implemented. | |||||
- Added sysctl controls. | |||||
-------------------- | |||||
Changes since 0.0.18 | |||||
-------------------- | |||||
- No patch for OFED is necessary for FreeBSD 11.0. | |||||
-------------------- | |||||
Changes since 0.0.11 | |||||
-------------------- | |||||
- Support for FreeBSD 11.0 connection is added. | |||||
- No patch for OFED is necessary for FreeBSD 10.3. | |||||
================================================================================ | |||||
Support | |||||
------- | |||||
For general information, go to the Intel support website at: | |||||
www.intel.com/support/ | |||||
or the Intel Wired Networking project hosted by Sourceforge at: | |||||
http://sourceforge.net/projects/e1000 | |||||
If an issue is identified with the released source code on a supported | |||||
kernel with a supported adapter, email the specific information related to the | |||||
issue to e1000-rdma@lists.sourceforge.net | |||||
================================================================================ | |||||
License | |||||
------- | |||||
This software is available to you under a choice of one of two | |||||
licenses. You may choose to be licensed under the terms of the GNU | |||||
General Public License (GPL) Version 2, available from the file | |||||
COPYING in the main directory of this source tree, or the | |||||
OpenFabrics.org BSD license below: | |||||
Redistribution and use in source and binary forms, with or | |||||
without modification, are permitted provided that the following | |||||
conditions are met: | |||||
- Redistributions of source code must retain the above | |||||
copyright notice, this list of conditions and the following | |||||
disclaimer. | |||||
- Redistributions in binary form must reproduce the above | |||||
copyright notice, this list of conditions and the following | |||||
disclaimer in the documentation and/or other materials | |||||
provided with the distribution. | |||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, | |||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF | |||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND | |||||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS | |||||
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN | |||||
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN | |||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | |||||
SOFTWARE. | |||||
================================================================================ | |||||
Trademarks | |||||
---------- | |||||
Intel and Itanium are trademarks or registered trademarks of Intel | |||||
Corporation or its subsidiaries in the United States and/or other countries. | |||||
* Other names and brands may be claimed as the property of others. | |||||