SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that allows the use of SCSI devices across infiniband. If you have infiniband storage, use can use SRP to acess the devices.
Ensure that your infiniband storage is presented to the host in question. Check your storage controller documentation. Ensure that the ib_srp kernel module is loaded and that the srptools package is installed.
srp_daemon is responsible for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented devices; this is a failsafe to prevent devices from being mounted by accident on the wrong hosts.
The srp_daemon config file /etc/srp_daemon.conf has a simply syntax, and is described in the srp_daemon(1) manpage. Each line in this file is a rule which can be either to allow connection or to disallow connection according to the first character in the line (a or d accordingly) and ID of the storage device.
You can determine the IDs of SRP devices presented to your hosts by running the ibsrpdm -c command.
# ibsrpdm -c id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050
Once we have the IDs of the devices, we can add them to /etc/srp_daemon.conf. You can also specify other srp related options for the target, such as max_cmd_per_lun and Max_sect. These are storage specific; check your vendor documentation for reccomended values.
Restart the srp_daemon and the storage target should now become visible; check the kernel log to see if the disk has been detected.
# This rule allows connection to our target a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535 # This rule disallows everything else d
In the example kernel log output the disk has been descovered as scsi device sdb.
scsi 3:0:0:1: Direct-Access IBM DCS9900 5.03 PQ: 0 ANSI: 5 sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB) sd 3:0:0:1: [sdb] Write Protect is off sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08 sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB) sd 3:0:0:1: [sdb] Write Protect is off sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08 sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA sdb:<6>scsi4 : SRP.T10:50001FF10005052A unknown partition table sd 3:0:0:1: [sdb] Attached SCSI disk sd 3:0:0:1: Attached scsi generic sg5 type 0
The newly detected SRP device can be treated as an other scsi device. If you have multiple infiniband adapters you can use multipath-tools on top of the SRP devices to protects against a network failure. If you are not using multipathed IO you can simply format the device as normal.