Building a 20TB FC SAN Target on Debian 12

To expose the disks at a block level, we have to install targetcli.
For this setup, we will expose the entire disk.

apt install targetcli-fb

targetcli is an interactive shell used to manage the SCSI targets, LUNs and ACLs between others.

The GitHub repository can be found here and the documentation can be found here (documentation does not work at the moment of writing this guide).

Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/>

With ls we can see all the options available

/> ls
o- / ............................................................................................ [...]
  o- backstores ................................................................................. [...]
  | o- block ..................................................................... [Storage Objects: 0]
  | o- fileio .................................................................... [Storage Objects: 0]
  | o- pscsi ..................................................................... [Storage Objects: 0]
  | o- ramdisk ................................................................... [Storage Objects: 0]
  o- iscsi ............................................................................... [Targets: 0]
  o- loopback ............................................................................ [Targets: 0]
  o- qla2xxx ............................................................................. [Targets: 0]
  o- vhost ............................................................................... [Targets: 0]
  o- xen-pvscsi .......................................................................... [Targets: 0]

To get the info of the connected QLE2560 card run qla2xxx/ info

/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWN types: naa
Allowed WWNs list: naa.21000024ff10d45c
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx

Now, to create a target use qla2xxx/ create naa.21000024ff10d45c
The naa.21000024ff10d45c has been obtained from the HBA adapter before installing it, or with the info command.

/> qla2xxx/ create naa.21000024ff10d45c
Created target naa.21000024ff10d45c.

For this guide a block storage object identified as LUN0 will be created and point tho the virtual array /dev/sda created with the LSI card. This will be done using the following command: backstores/block create lun0 /dev/sda

/> backstores/block create lun0 /dev/sda
Created block storage object lun0 using /dev/sda.

Create a new LUN for the interface, attached to a previously defined storage object. The storage object now shows up under the /backstores configuration node as activated.

/> ls
o- / ............................................................................................ [...]
  o- backstores ................................................................................. [...]
  | o- block ..................................................................... [Storage Objects: 1]
  | | o- lun0 ............................................ [/dev/sda (929.5GiB) write-thru deactivated]
  | |   o- alua ...................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp .......................................... [ALUA state: Active/optimized]
  | o- fileio .................................................................... [Storage Objects: 0]
  | o- pscsi ..................................................................... [Storage Objects: 0]
  | o- ramdisk ................................................................... [Storage Objects: 0]
  o- iscsi ............................................................................... [Targets: 0]
  o- loopback ............................................................................ [Targets: 0]
  o- qla2xxx ............................................................................. [Targets: 1]
  | o- naa.21000024ff10d45c ................................................................ [gen-acls]
  |   o- acls ............................................................................... [ACLs: 0]
  |   o- luns ............................................................................... [LUNs: 0]
  o- vhost ............................................................................... [Targets: 0]
  o- xen-pvscsi .......................................................................... [Targets: 0]

To create the new LUN for the qla2xxx target use the following command: qla2xxx/naa.210000e08b943494/luns create /backstores/block/lun0

/> qla2xxx/naa.21000024ff10d45c/luns create /backstores/block/lun0
Created LUN 0.

Now, create an ACL (Access Control List) to define the resources each initiator can access. The default behavior is to auto-map existing LUNs to the ACL.
Use the command qla2xxx/naa.21000024ff10d45c/acls create XXXX where XXXX is the initiator's WWN. This will prevent other initiators from accessing the LUN. If the lun0 filesystem does not support clustering, having multiple HBAa accessing the same LUN will result in data loss and critical failure.

/> qla2xxx/naa.21000024ff10d45c/acls create 21000024ff10d508
Created Node ACL for naa.21000024ff10d508
Created mapped LUN 0.

Use ls command to see the final configuration

/> ls
o- / ............................................................................................ [...]
  o- backstores ................................................................................. [...]
  | o- block ..................................................................... [Storage Objects: 1]
  | | o- lun0 .............................................. [/dev/sda (929.5GiB) write-thru activated]
  | |   o- alua ...................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp .......................................... [ALUA state: Active/optimized]
  | o- fileio .................................................................... [Storage Objects: 0]
  | o- pscsi ..................................................................... [Storage Objects: 0]
  | o- ramdisk ................................................................... [Storage Objects: 0]
  o- iscsi ............................................................................... [Targets: 0]
  o- loopback ............................................................................ [Targets: 0]
  o- qla2xxx ............................................................................. [Targets: 1]
  | o- naa.21000024ff10d45c ................................................................ [gen-acls]
  |   o- acls ............................................................................... [ACLs: 1]
  |   | o- naa.21000024ff10d508 ...................................................... [Mapped LUNs: 1]
  |   |   o- mapped_lun0 ....................................................... [lun0 block/lun0 (rw)]
  |   o- luns ............................................................................... [LUNs: 1]
  |     o- lun0 ............................................ [block/lun0 (/dev/sda) (default_tg_pt_gp)]
  o- vhost ............................................................................... [Targets: 0]
  o- xen-pvscsi .......................................................................... [Targets: 0]

Finally, save using saveconfig command.

/> saveconfig
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

Use exit to exit targetcli command line.

Once the fiber cable of the initiator is plugged to the target (Point to Point Architecture), a bus reset must be launched. This will scan the interconnect and will cause device addition or removal.

Use the command echo 1 > /sys/class/fc_host/hostN/issue_lip replacing hostN with the host number of your system.

echo 1 > /sys/class/fc_host/host8/issue_lip

Now, by using the dmesg command, you can verify that the operation was successful. The new disk will be visible by running fdisk -l.

[   75.037265] qla2xxx [0000:04:00.0]-500a:8: LOOP UP detected (8 Gbps).
[  177.851132] Rounding down aligned max_sectors from 4294967295 to 4294967288
[  177.851188] db_root: cannot open: /etc/target
[ 1508.390714] scsi 8:0:0:0: Direct-Access     LIO-ORG  lun0             4.0  PQ: 0 ANSI: 6
[ 1508.393960] sd 8:0:0:0: Attached scsi generic sg3 type 0
[ 1508.394125] sd 8:0:0:0: [sdd] 1949216768 512-byte logical blocks: (998 GB/929 GiB)
[ 1508.394210] sd 8:0:0:0: [sdd] Write Protect is off
[ 1508.394214] sd 8:0:0:0: [sdd] Mode Sense: 43 00 10 08
[ 1508.394346] sd 8:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 1508.394423] sd 8:0:0:0: [sdd] Preferred minimum I/O size 512 bytes
[ 1508.394427] sd 8:0:0:0: [sdd] Optimal transfer size 327680 bytes
[ 1508.437028] sd 8:0:0:0: [sdd] Attached SCSI disk

Disk /dev/sdd: 929.46 GiB, 997998985216 bytes, 1949216768 sectors
Disk model: lun0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 327680 bytes

This disk has no partitions yet.
To partition the disk use fdisk /dev/sdXX tool. Where /dev/sdXX is the path of the disk on your system.

fdisk /dev/sdd
Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0xc639a019.

Command (m for help):  n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-1949216767, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-1949216767, default 1949216767):

Created a new partition 1 of type 'Linux' and of size 929.5 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Now the new partition has to be formatted using mkfs tool. For this guide Ext4 filesystem has been selected.

mkfs.ext4 /dev/sdd1
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 243651840 4k blocks and 60915712 inodes
Filesystem UUID: c30ac810-eb26-492b-8c5c-1f004fa77d64
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Create a directory on the /mnt/ path called SAN_LUN0

mkdir /mnt/SAN_LUN0

Finally, mount it using mount command

mount /dev/sdd1 /mnt/SAN_LUN0/

For the requirements of this guide we will transfer ownership of the disk to the default user nexus in this case.

chown nexus:nexus /mnt/SAN_LUN0

Now the FC SAN is set it is time to test the performance on RAID 5 with three HDDs and three SSDs respectively.
4 GiB files has been chosen to exhaust the LSI cache and avoid be performing test on the cache itself.

If you would like to see other performance comparisons contact me at antonimercer@lthjournal.com

During the development of this project, these two books have proved to be really helpful.
While the IBM Redbooks' Introduction to Storage Area Networks does not get into the technical details, it provides an introduction to the technology and explains the associated hardware. It begins at a very basic level by presenting the problems that SAN networks address and the various options for storage connectivity. This book will give you an understanding of what this project aims to solve, while Storage Area Network Essentials delves deeper into the specifics of constructing one.