High performance SCST iSCSI Target on Linux software Raid: Difference between revisions

From Alpine Linux
(Using existing template for WIP/work in progress)
Line 1: Line 1:
'''This is a WIP'''
{{Draft}}


== Introduction ==
== Introduction ==

Revision as of 16:03, 6 January 2011

This material is work-in-progress ...

Do not follow instructions here until this notice is removed.
(Last edited by Mhavela on 6 Jan 2011.)

Introduction

This HOW-TO is focusing on performance. This is why I made some decisions targeted on performance instead of security (like you are used in Alpine Linux). This means we are not using grsec and not using a firewall. I presume you will take security actions on another level.

To get started, you can download a boot cd here:

http://alpine.nethq.org/distfiles/alpine-scst-110106-x86_64.iso

Content

  • Vanilla kernel with SCST (linux-scst)
  • Linux software raid and raid level (mdadm)
  • Volume Management (LVM2)
  • SCST administration (scstadmin)
  • Linux raid monitoring

Vanilla Linux kernel with SCST patches

The default Linux kernel will provide support for iSCSI. The biggest issues with this implementation is it operates in userspace. SCST iSCSI will run in kernel and this is one of the reasons why it preforms much better. SCST performance depends on specific patches which need to be applied to the kernel. This is why we created a separate kernel just for SCST usage. SCST modules are already included by default so there is no need for a separate module package to be installed.

P.S. We only provide an x86_64 kernel for SCST because it will perform better on 64bit systems.

Linux software raid

In my personal setup i have 4 pieces of WD RE4 1TB drives which i want to use in the best performance raid level with redundancy. According to many mailing lists and opinion of the Linux raid author RAID10 with layout f2 (far) seems to preform best while still having redundancy. Please remember with RAID10 50% of your hard disk space will go to redundancy, but performance is almost the same as RAID0 (stripe).

For most up-to-date information regarding Linux software raid: https://raid.wiki.kernel.org/index.php/Overview

RAID10 has multiple layout types. f(far)2 in tests seem to preform the best. Please see above link for references.

mdadm -v --create /dev/md0 --level=raid10 --layout=f2 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

I am not using partitions on my disks, although there are reasons to use partitions, see here:

https://raid.wiki.kernel.org/index.php/Partition_Types

You can now monitor your raid (re)building: cat /proc/mdstat

By default, the rebuild speed will be set and can be checked and changed here:

cat /proc/sys/dev/raid/speed_limit_max
cat /proc/sys/dev/raid/speed_limit_min

make sure we have raid10 module loaded at boot

echo raid10 >> /etc/modules

When you are happy with your raid configuration, save its information to mdamd.conf file

mdadm --detail --scan >> /etc/mdadm.conf

It should display something like

ARRAY /dev/md0 metadata=1.2 name=scst:0 UUID=71fc93b8:3fef0057:9f5feec1:7d9e57e8

When you are ready with your raid setup and its functioning, you will need to make sure its starting at boot time

rc-update add mdadm-raid default

Monitor software raid

Linux software raid can be monitored with mdadm daemon option. Alpine Linux includes an initd script which can invoke the daemon

/etc/init.d/mdadm

It will be default monitor the array's defined in mdadm.conf. To receive email notifications about array issues, we need to provide our email address inside mdadm.conf:

MAILADDR me@inter.net

Because mdadm cannot send email itself, we need to setup an sendmail (replacement) program.

apk add ssmtp

Ssmtp can be configured by editing /etc/ssmtp/ssmtp.conf

You can monitor messages (syslog) for actions invoked by mmdadm.

If you have your own monitoring system active, you can also let mdadm issue a script and notify it.

SCST and iSCSI management

When starting with SCST management we need to have the SCST framework (kernel module) and the iSCSI kernel module loaded

modprobe scst scst_vdisk iscsi_scst

After these are loaded we can start the iSCSI deamon

/etc/init.d/iscsi-scst start

This command will return some information based on the current configuration located in: /etc/scst.conf

The basic config when only having iSCSI loaded is:

TARGET_DRIVER iscsi {
        enabled 0
}

We start add adding a target to the correct target driver

scstadmin -add_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi

The config at this point should be:

TARGET_DRIVER iscsi {
        enabled 0

        TARGET iqn.2010-12.org.alpinelinux:tgt {
                enabled 0
        }
}

Now that we have a target in our configuration we need to add a device:

scstadmin -open_dev disk01 -handler vdisk_fileio -attributes filename=/mnt/array1/disk01,nv_cache=1

The config at this point should be:

HANDLER vdisk_fileio {
       DEVICE disk01 {
               t10_dev_id "disk01 b8ceed65"
               usn b8ceed65

               filename /mnt/array1/disk01
               nv_cache 1
       }
}

TARGET_DRIVER iscsi {
       enabled 0

       TARGET iqn.2010-12.org.alpinelinux:tgt {
               enabled 0
       }
}

To add the device to the target we need to specify which LUN it will be. (we always need to start with 0)

scstadmin -add_lun 0 -driver iscsi -target iqn.2010-12.org.alpinelinux:tgt -device disk01

The config at this point should be:

HANDLER vdisk_fileio {
       DEVICE disk01 {
               t10_dev_id "disk01 b8ceed65"
               usn b8ceed65

               filename /mnt/array1/disk01
               nv_cache 1
       }
}

TARGET_DRIVER iscsi {
       enabled 0

       TARGET iqn.2010-12.org.alpinelinux:tgt {
               enabled 0

               LUN 0 disk01
       }
}

This is the default minimum configuration for a working iSCSI setup. We now need to activate it:

scstadmin -enable_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi

The config at this point should be:

HANDLER vdisk_fileio {
       DEVICE disk01 {
               t10_dev_id "disk01 b8ceed65"
               usn b8ceed65

               filename /mnt/array1/disk01
               nv_cache 1
       }
}

TARGET_DRIVER iscsi {
       enabled 0

       TARGET iqn.2010-12.org.alpinelinux:tgt {
               rel_tgt_id 1
               enabled 1

               LUN 0 disk01
       }
}

NOTE: I'm currently not sure why the target driver is not activated by the above command. Looks like the only option in scstadmin which could do it. You will need to manually set it to 1 in the config and restart iscsi-scst.

To disable the target

scstadmin -disable_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi

To remove the lun

scstadmin -rem_lun 0 -driver iscsi -target iqn.2010-12.org.alpinelinux:tgt -device disk01

To close a device

scstadmin -close_dev disk01 -handler vdisk_fileio

To remove a target

scstadmin -rem_target iqn.2010-12.org.alpinelinux:tgt -driver iscsi