Sockets Direct Protocol (SDP) is a network protocol which provides an RDMA accelerated alternative to TCP over infiniband networks. OFED provides an LD_PRELOADable library (libsdp.so) which allows programs which use TCP to use the more efficient SDP protocol instead. The use of an LD_PRELOADable libary means that the switch in protocol is transparent, and does not require the application to be recompiled.
SDP used IPoIB for address resolution, so you must configure IPoIB before using SDP.
You should also ensure the ib_sdp kernel module is installed.
You can use libsdp in two ways; you can either manually LD_PRELOAD the library whilst invoking your application, or create a config file which specifies which applications will use SDP.
To manually LD_PRELOAD a library, simply set the LD_PRELOAD variable before invoking your application.
LD_PRELOAD=libsdp.so ./path/to/your/application ...If you which to choose which programs will use SDP you can edit /etc/sdp.conf and specify which programs, ports and addresses are eligible for use.
The following example shows how to use libsdp to make the TCP benchmarking application, netpipe, use SDP rather than TCP. NodeA is the server and NodeB is the client. IPoIB is configured on both nodes, and NodeA's IPoIB address is 10.0.0.1
Install netpipe on both nodes.
aptitude install netpipe-tcp
First, run the netpipe benchmark over TCP in order to obtain a baseline number.
nodeA# NPtcp nodeB# NPtcp -h 10.0.0.1 Send and receive buffers are 16384 and 87380 bytes (A bug in Linux doubles the requested buffer sizes) Now starting the main loop 0: 1 bytes 2778 times --> 0.22 Mbps in 34.04 usec 1: 2 bytes 2937 times --> 0.45 Mbps in 33.65 usec 2: 3 bytes 2971 times --> 0.69 Mbps in 33.41 usec <snip> 121: 8388605 bytes 3 times --> 2951.89 Mbps in 21680.99 usec 122: 8388608 bytes 3 times --> 3008.08 Mbps in 21276.00 usec 123: 8388611 bytes 3 times --> 2941.76 Mbps in 21755.66 usec
Now repeat the test, but force netpipe to use SDP rather than TCP.
You should see a significant increase in performance when using SDP.
nodeA# LD_PRELOAD=libsdp.so NPtcp nodeB# LD_PRELOAD=libsdp.so NPtcp -h 10.0.0.1 Send and receive buffers are 16384 and 87380 bytes (A bug in Linux doubles the requested buffer sizes) Now starting the main loop 0: 1 bytes 9765 times --> 1.45 Mbps in 5.28 usec 1: 2 bytes 18946 times --> 2.80 Mbps in 5.46 usec 2: 3 bytes 18323 times --> 4.06 Mbps in 5.63 usec <snip> 121: 8388605 bytes 5 times --> 7665.51 Mbps in 8349.08 usec 122: 8388608 bytes 5 times --> 7668.62 Mbps in 8345.70 usec 123: 8388611 bytes 5 times --> 7629.04 Mbps in 8389.00 usec