The iWARP Connection Manager (CM) on FreeBSD creates a TCP socket to
represent an iWARP endpoint when the connection is over TCP. For
servers the current approach is to invoke create_listen callback for
each iWARP RNIC registered with the CM. This doesn't work too well for
INADDR_ANY because a listen on any TCP socket already notifies all
hardware TOEs/RNICs of the new listener. This patch fixes the server
side of things for FreeBSD. We've tried to keep all these modifications
in the iWARP/TCP specific parts of the OFED infrastructure as much as
possible.
Submitted by: Krishnamraju Eraparaju @ Chelsio
Substantial design inputs from: Steve Wise @ Open Grid Computing.
Here are Steve's raw notes which outline the approach taken in this patch.
So, here is what I think we can do that minimally impacts the cma.c code:
- we remove the create_listen()/destroy_listen inter-module API call between
the IWCM and the iWARP drivers. listening and accepting will be done 100% in
the IWCM.
- cma.c/cma_listen_on_dev() will copy the parent socket pointer to the new
device-specific cm_id it creates. This allows the socket bound to INADDR_ANY
to be passed to the IWCM. This is one change to the cma.c common code we need,
but it's a one-liner.
- Assume an app calls rdma_bind_addr(INADDR_ANY). So a socket bound to
INADDR_ANY is allocated at that time via a call to
cma_get_port()->cma_get_tcp_port(). If the app doesn't call rdma_bind_addr(),
but calls rdma_listen() directly, rdma_listen() calls
rdma_bind_addr() so either way we end up with the cm_id having a socket pointer
that is bound to INADDR_ANY. So here's the call chain for an INADDR_ANY
listen. ulp->rdma_listen()->cma_listen_on_all()->cma_listen_on_dev() for each
device->rdma_listen() with a new cm_id that is attached to the device -and now
has a pointer to the socket created in step 2. rdma_listen(cm_id bound to a
specific device) calls cma_iw_listen(). cma_iw_listen() creates a new iw_cm_id
passing the socket pointer, and then calls
iw_cm_listen() which is in iw_cm.c
So now iw_cm_listen() has either a pointer to a socket bound to a specific
address (as is currently working) or the socket bound to INADDR_ANY. In either
case, I propose we change this code and any supporting code to have
iw_cm_listen() handle all listening sockets. iw_cm_listen() can call
solisten() and handle newly accepted sockets. In the case of multiple iwarp
devices and a listening endpoint bound to INADDR_ANY, iw_cm_listen() will get
called with device specific iw_cm_Id's that all have the same socket pointer.
All it has to do is only call solisten() once, and then process accepted
sockets and deliver to the appropriate iw_cm_id in its list of cm_id's using
this INADDR_ANY socket.
- Add a new intra-module API between IWCM and iWARP drivers to pass newly
accepted sockets to the driver. The driver does what it previously did when it
got a new socket from its listening socket.
I think this will work with only a small change to the common cma.c code...
noodle on this and ask questions... :)