|
|
| (40 intermediate revisions by 5 users not shown) |
| Line 1: |
Line 1: |
| {{Draft}}
| | #REDIRECT [[Linux iSCSI Target (TCM)]] |
| | |
| == Introduction ==
| |
| | |
| This HOW-TO is focusing on performance. This is why I made some decisions targeted on performance instead of security (like you are used in Alpine Linux).
| |
| This means we are not using grsec and not using a firewall. I presume you will take security actions on another level.
| |
| | |
| To get started, you can download a boot cd here:
| |
| | |
| http://alpine.nethq.org/distfiles/alpine-scst-110106-x86_64.iso
| |
| | |
| == Vanilla Linux kernel with SCST patches ==
| |
| | |
| The default Linux kernel will provide support for iSCSI. The problem with this implementation is it operates in user-space. SCST iSCSI will run in kernel-space and this is one of the reasons why it preforms much better. SCST performance depends on specific patches which need to be applied to the kernel. This is why we created a separate kernel just for SCST usage. SCST modules are already included by default so there is no need for a separate module package to be installed.
| |
| | |
| P.S. We only provide an x86_64 kernel for SCST because it will perform better on 64bit systems.
| |
| | |
| == Linux software raid ==
| |
| | |
| In my personal setup i have 4 pieces of WD RE4 1TB drives which i want to use in the best performance raid level with redundancy.
| |
| According to many mailing lists and opinion of the Linux raid author RAID10 with layout f2 (far) seems to preform best while still having redundancy.
| |
| Please remember with RAID10 50% of your hard disk space will go to redundancy, but performance is almost the same as RAID0 (stripe).
| |
| | |
| For most up-to-date information regarding Linux software raid: https://raid.wiki.kernel.org/index.php/Overview
| |
| | |
| RAID10 has multiple layout types. f(far)2 in tests seem to preform the best. Please see above link for references.
| |
| | |
| mdadm -v --create /dev/md0 --level=raid10 --layout=f2 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
| |
| | |
| I am not using partitions on my disks, although there are reasons to use partitions, see here:
| |
| | |
| https://raid.wiki.kernel.org/index.php/Partition_Types
| |
| | |
| You can now monitor your raid (re)building: cat /proc/mdstat
| |
| | |
| By default, the rebuild speed will be set and can be checked and changed here:
| |
| | |
| cat /proc/sys/dev/raid/speed_limit_max
| |
| cat /proc/sys/dev/raid/speed_limit_min
| |
| | |
| make sure we have raid10 module loaded at boot
| |
| | |
| echo raid10 >> /etc/modules
| |
| | |
| When you are happy with your raid configuration, save its information to mdamd.conf file
| |
| | |
| mdadm --detail --scan >> /etc/mdadm.conf
| |
| | |
| It should display something like
| |
| | |
| ARRAY /dev/md0 metadata=1.2 name=scst:0 UUID=71fc93b8:3fef0057:9f5feec1:7d9e57e8
| |
| | |
| When you are ready with your raid setup and its functioning, you will need to make sure its starting at boot time
| |
| | |
| rc-update add mdadm-raid default
| |
| | |
| == Monitor software raid ==
| |
| | |
| Linux software raid can be monitored with mdadm daemon option. Alpine Linux includes an initd script which can invoke the daemon | |
| | |
| /etc/init.d/mdadm
| |
| | |
| It will be default monitor the array's defined in mdadm.conf. To receive email notifications about array issues, we need to provide our email address inside mdadm.conf:
| |
| | |
| MAILADDR me@inter.net
| |
| | |
| Because mdadm cannot send email itself, we need to setup an sendmail (replacement) program.
| |
| | |
| apk add ssmtp
| |
| | |
| Ssmtp can be configured by editing /etc/ssmtp/ssmtp.conf
| |
| | |
| You can monitor messages (syslog) for actions invoked by mmdadm.
| |
| | |
| If you have your own monitoring system active, you can also let mdadm issue a script and notify it.
| |
| | |
| == Disk & Volume management with SCST ==
| |
| | |
| Volume management can be an interesting addition to your block device(s) but it does add is an extra layer between you block devices and SCST.
| |
| If you are in need of any of the features provided by LVM then go ahead and use it, I didn't add it to my array cause i don't think i will need it.
| |
| In my above RAID10 setup, md will provide me an 2TB block device md0. SCST provides 2 ways to access your disk subsystem, BLOCKIO and FILEIO.
| |
| | |
| BLOCKIO has access directly to the block device md0 (without extra layers) but FILEIO uses a filesystem on top of the block device and will use regular files. Altough the extra layer would seem to be bad but it actually isn't. The filesystem will provide a caching layer and in some situations you will see improved performance.
| |
| | |
| Some speed tests:
| |
| | |
| http://scst.sourceforge.net/vl_res.txt
| |
| | |
| When using FILEIO we need to create a filesystem. To create an XFS filesystem we need xfsprogs installed
| |
| | |
| apk add xfsprogs
| |
| | |
| We also need XFS support in our kernel
| |
| | |
| modprobe xfs
| |
| | |
| And we need it next time we boot
| |
| | |
| echo xfs >> /etc/modules
| |
| | |
| XFS will automatically choose the correct settings for the target block device
| |
| | |
| mkfs.xfs /dev/md0
| |
| | |
| Our filesystem should be ready to be mounted
| |
| | |
| mkdir -p /mnt/array1 && mount /dev/md0 /mnt/array1
| |
| | |
| Because of FILEIO we need files on our filesystem which will act as iSCSI disks. We will create them with dd
| |
| | |
| dd if=/dev/zero of=/mnt/array1/disk01 bs=512k count=100000
| |
| | |
| You can also user sparse files which are instantly created (no need to wait for every bit to be written to the filesystem).
| |
| | |
| More info about sparse files here
| |
| | |
| http://en.wikipedia.org/wiki/Sparse_file
| |
| | |
| == SCST and iSCSI management ==
| |
| | |
| SCST is managed by the sysfs filesystem. You can create your own scripts to control it, or use the included scstadmin like I will show here.
| |
| | |
| For a detailed overview of the sysfs filesystem please check here:
| |
| | |
| http://lwn.net/Articles/378658/
| |
| | |
| When starting with SCST management we need to have the SCST framework (kernel module) and the iSCSI kernel module loaded
| |
| | |
| modprobe scst scst_vdisk iscsi_scst
| |
| | |
| After these are loaded we can start the iSCSI deamon
| |
| | |
| /etc/init.d/iscsi-scst start
| |
| | |
| This command will return some information based on the current configuration located in: /etc/scst.conf
| |
| | |
| The basic config when only having iSCSI loaded is:
| |
| | |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
| }
| |
| | |
| === Adding a target ===
| |
| | |
| We start add adding a target to the correct target driver
| |
| | |
| scstadmin -add_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi
| |
| | |
| The config at this point should be:
| |
| | |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
|
| |
| TARGET iqn.2010-12.org.alpinelinux:tgt {
| |
| enabled 0
| |
| }
| |
| }
| |
| | |
| === Adding a device ===
| |
| | |
| Now that we have a target in our configuration we need to add a device:
| |
| | |
| scstadmin -open_dev disk01 -handler vdisk_fileio -attributes filename=/mnt/array1/disk01,nv_cache=1
| |
| | |
| The config at this point should be:
| |
| | |
| HANDLER vdisk_fileio {
| |
| DEVICE disk01 {
| |
| t10_dev_id "disk01 b8ceed65"
| |
| usn b8ceed65
| |
|
| |
| filename /mnt/array1/disk01
| |
| nv_cache 1
| |
| }
| |
| }
| |
|
| |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
|
| |
| TARGET iqn.2010-12.org.alpinelinux:tgt {
| |
| enabled 0
| |
| }
| |
| }
| |
| | |
| === Adding a LUN ===
| |
| | |
| To add the device to the target we need to specify which LUN it will be. (we always need to start with 0)
| |
| | |
| scstadmin -add_lun 0 -driver iscsi -target iqn.2010-12.org.alpinelinux:tgt -device disk01
| |
| | |
| The config at this point should be:
| |
| | |
| HANDLER vdisk_fileio {
| |
| DEVICE disk01 {
| |
| t10_dev_id "disk01 b8ceed65"
| |
| usn b8ceed65
| |
|
| |
| filename /mnt/array1/disk01
| |
| nv_cache 1
| |
| }
| |
| }
| |
|
| |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
|
| |
| TARGET iqn.2010-12.org.alpinelinux:tgt {
| |
| enabled 0
| |
|
| |
| LUN 0 disk01
| |
| }
| |
| }
| |
| | |
| === Enable the target ===
| |
| | |
| This is the default minimum configuration for a working iSCSI setup. We now need to activate it:
| |
| | |
| scstadmin -enable_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi
| |
| | |
| The config at this point should be:
| |
| | |
| HANDLER vdisk_fileio {
| |
| DEVICE disk01 {
| |
| t10_dev_id "disk01 b8ceed65"
| |
| usn b8ceed65
| |
|
| |
| filename /mnt/array1/disk01
| |
| nv_cache 1
| |
| }
| |
| }
| |
|
| |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
|
| |
| TARGET iqn.2010-12.org.alpinelinux:tgt {
| |
| rel_tgt_id 1
| |
| enabled 1
| |
|
| |
| LUN 0 disk01
| |
| }
| |
| }
| |
| | |
| === SCST security settings ===
| |
| | |
| The latest version of SCST has changes its security configuration. Previously you should edit files inside your etc directory like initiators.allow but since SysFS its it controlled with SysFS.
| |
| iSCSI daemon can be configured to be only listening to one interface, but in the cause of MPIO you would have multiple interfaces to listen on. To make sure only your initiators can see your devices we can add its IP as an attribute to the target
| |
| | |
| scstadmin -add_tgt_attr iqn.2010-12.org.alpinelinux:tgt -driver iscsi -attributes allowed_portal=192.168.1.1
| |
| | |
| IP wildcards are specified with a question mark "?"
| |
| | |
| HANDLER vdisk_fileio {
| |
| DEVICE disk01 {
| |
| t10_dev_id "disk01 b8ceed65"
| |
| usn b8ceed65
| |
|
| |
| filename /mnt/array1/disk01
| |
| nv_cache 1
| |
| }
| |
| }
| |
|
| |
| TARGET_DRIVER iscsi {
| |
| enabled 0
| |
|
| |
| TARGET iqn.2010-12.org.alpinelinux:tgt {
| |
| rel_tgt_id 1
| |
| enabled 1
| |
| allowed_portal 192.168.1.1
| |
|
| |
| LUN 0 disk01
| |
| }
| |
| }
| |
| | |
| | |
| === Enable the iSCSI driver ===
| |
| | |
| Our configuration should be ok and the target is enabled. If you need to set any security groups or other settings please do them now. When you are ready we can activate the iSCSI driver:
| |
| | |
| scstadmin -set_drv_attr iscsi -attributes enabled=1
| |
| | |
| === Reverting your changes ===
| |
| | |
| After this command, your iSCSI disk should be available in your initiator. You can use the same command to disable the target and take it offline:
| |
| | |
| scstadmin -set_drv_attr iscsi -attributes enabled=0
| |
| | |
| Some other command to break down the SCST config piece by piece:
| |
| | |
| To disable the target
| |
| | |
| scstadmin -disable_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi
| |
| | |
| To remove the lun
| |
| | |
| scstadmin -rem_lun 0 -driver iscsi -target iqn.2010-12.org.alpinelinux:tgt -device disk01
| |
| | |
| To close a device
| |
| | |
| scstadmin -close_dev disk01 -handler vdisk_fileio
| |
| | |
| To remove a target
| |
| | |
| scstadmin -rem_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi
| |