These install/howto instructions are for CentOS 5.x and Dell MD3000i. You can easily modify these slightly for your specific iSCSI disk.
iSCSI/Multipath/LVM
iSCSI/Multipath Installation
Install iSCSI initiator utilities
yum install iscsi-initiator-utils
Install multipath device mapper
yum install device-mapper-multipath
Configure multipath, iscsi, and iscsid to start on startup. Check first to make sure they are not already set to start on startup. If they are set to start you will see something like below, if they are not you will not see anything output listed as “on”.
/sbin/chkconfig --list | egrep -i "iscsi|iscsid|multipathd"
iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
To configure multipathd to start on startup do the following. iSCSI(d)already have this defined in their init script.
/sbin/chkconfig multipathd on
iSCSI Configuration
Edit /etc/iscsi/iscsid.conf and make sure it is identical to this config.
**I need to verify this iscsid.conf but for now this is what I have logged as a working configs from years back.
#
# Open-iSCSI default configuration.
# Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf
#
# Note: To set any of these values for a specific node/session run
# the iscsiadm --mode node --op command for the value. See the README
# and man page for iscsiadm for details on the --op command.
#
################
# iSNS settings
################
# Address of iSNS server
#isns.address = 192.168.0.1
#isns.port = 3205
#############################
# NIC/HBA and driver settings
#############################
# open-iscsi can create a session and bind it to a NIC/HBA.
# To set this up see the example iface config file.
#*****************
# Startup settings
#*****************
# To request that the iscsi initd scripts startup a session set to "automatic".
# node.startup = automatic
#
# To manually startup the session set to "manual". The default is automatic.
node.startup = automatic
# *************
# CHAP Settings
# *************
# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP
# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = username
#node.session.auth.password = password
# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = username_in
#node.session.auth.password_in = password_in
# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
#discovery.sendtargets.auth.authmethod = CHAP
# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
#discovery.sendtargets.auth.username = username
#discovery.sendtargets.auth.password = password
# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = username_in
#discovery.sendtargets.auth.password_in = password_in
# ********
# Timeouts
# ********
#
# See the iSCSI REAME's Advanced Configuration section for tips
# on setting timeouts when using multipath or doing root over iSCSI.
#
# To specify the length of time to wait for session re-establishment
# before failing SCSI commands back to the application when running
# the Linux SCSI Layer error handler, edit the line.
# The value is in seconds and the default is 120 seconds.
node.session.timeo.replacement_timeout = 120
# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 15
# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 15
# Time interval to wait for on connection before sending a ping.
node.conn[0].timeo.noop_out_interval = 5
# To specify the time to wait for a Nop-out response before failing
# the connection, edit this line. Failing the connection will
# cause IO to be failed back to the SCSI layer. If using dm-multipath
# this will cause the IO to be failed to the multipath layer.
node.conn[0].timeo.noop_out_timeout = 5
#******
# Retry
#******
# To speficy the number of times iscsiadm should retry a login
# to the target when we first login, modify the following line.
# The default is 4. Valid values are any integer value. This only
# affects the initial login. Setting it to a high value can slow
# down the iscsi service startup. Setting it to a low value can
# cause a session to not get logged into, if there are distuptions
# during startup or if the network is not ready at that time.
node.session.initial_login_retry_max = 4
################################
# session and device queue depth
################################
# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 128
# To control the device's queue depth set node.session.queue_depth
# to a value between 1 and 128. The default is 32.
node.session.queue_depth = 32
#***************
# iSCSI settings
#***************
# To enable R2T flow control (i.e., the initiator must wait for an R2T
# command before sending any data), uncomment the following line:
#
#node.session.iscsi.InitialR2T = Yes
#
# To disable R2T flow control (i.e., the initiator has an implied
# initial R2T of "FirstBurstLength" at offset 0), uncomment the following line:
#
# The defaults is No.
node.session.iscsi.InitialR2T = No
#
# To disable immediate data (i.e., the initiator does not send
# unsolicited data with the iSCSI command PDU), uncomment the following line:
#
#node.session.iscsi.ImmediateData = No
#
# To enable immediate data (i.e., the initiator sends unsolicited data
# with the iSCSI command packet), uncomment the following line:
#
# The default is Yes
node.session.iscsi.ImmediateData = Yes
# To specify the maximum number of unsolicited data bytes the initiator
# can send in an iSCSI PDU to a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144
node.session.iscsi.FirstBurstLength = 262144
# To specify the maximum SCSI payload that the initiator will negotiate
# with the target for, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the defauls it 16776192
node.session.iscsi.MaxBurstLength = 16776192
# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 131072
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072
# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target during a discovery session, edit the
# following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 32768
#
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of enabling the checking, uncommen
# the following lines (Data digests are not supported and on ppc/ppc64
# both header and data digests are not supported.):
#node.conn[0].iscsi.HeaderDigest = CRC32C,None
#
# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of disabling the checking,
# uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = None,CRC32C
#
# To enable CRC32C digest checking for the header and/or data part of
# iSCSI PDUs, uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C
#
# To disable digest checking for the header and/or data part of
# iSCSI PDUs, uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = None
#
# The default is to never use DataDigests and to allow the target to control
# the setting of the HeaderDigest checking with the initiator requesting
# a preference of disabling the checking.
#***************
# iSCSI Multipath settings
#***************
# Specify the use of having multiple network paths to the same target.
# Valid values are
# no - there is no multipathing. Establish at most one session to the
# target and use all portals for failover, if failover is not disabled.
# portal - establish a session to the target using each of the n/w portals to
# the target, and bind each session to a different host SCSI address.
# No portal failover is possible since each session has only one portal.
# portalgroup - establish a session to one portal from each portal group that
# reaches the target and bind each session to a different host SCSI
# address. If a portal group has more than one portal, portal failover
# can occur between that group's portals if failover is not disabled.
Multipath = portalgroup
Edit /etc/iscsi/initiatorname.iscsi and add the alias name you created (in the MD3000i for this host) to this config.
InitiatorAlias=iqn.SERVER_HOSTNAME
Discover the iSCSI target.
iscsiadm -m discovery -t sendtargets -p 192.168.130.101
192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
Login to the iSCSI target.
iscsiadm -m node -l
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.101,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.102,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.101,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.102,3260]
Verify you are logged in to the iSCSI target and have a session
iscsiadm -m session
tcp: [5] 192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [6] 192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [7] 192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [8] 192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
Logout of the iSCSI target (if you ever need to).
iscsiadm -m node -u
Logout session [sid: 9, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.101,3260]
Logout session [sid: 10, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.102,3260]
Logout session [sid: 11, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.101,3260]
Logout session [sid: 12, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.102,3260]
Verify that the iSCSI target has you as a host and has disks mapped to you.
You will want to ignore any Disk the doesn’t contain a valid partition table
You can also check /var/log/dmesg to see similar but more verbose info, but is not covered here.
fdisk -l
Disk /dev/sda: 72.7 GB, 72746008576 bytes
255 heads, 63 sectors/track, 8844 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 8844 70935007+ 8e Linux LVM
Disk /dev/sdb: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdf: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes
Disk /dev/sdf doesn't contain a valid partition table
Disk /dev/sdg: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes
Disk /dev/sdg doesn't contain a valid partition table
Disk /dev/sdh: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes
Disk /dev/sdh doesn't contain a valid partition table
Disk /dev/sdi: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes
Disk /dev/sdi doesn't contain a valid partition table
Disk /dev/dm-3: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-3 doesn't contain a valid partition table
Multipath Configuration
Now that we have the modules compiled and installed, we need to setup a configuration file so that the multipath modules/daemon will choose the correct devices.
Edit /etc/multipath.conf‘ by commenting out everything that is not already commented out. Then insert the following
# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated
defaults {
udev_dir /dev
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^sda"
devnode "^sda[0-9]"
device {
vendor DELL
product "PERC|Universal|Virtual"
}
}
devices {
device {
vendor DELL
product MD3000i
hardware_handler "1 rdac"
path_checker rdac
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
failback immediate
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
}
}
multipaths {
mulitpath {
device {
vendor DELL
product MD3000i
}
}
}
Multipath Discovery
Now we have a configuration and modules, lets clear multipath database and re-establish the paths.
To flush the database use -F and the restart the deamon.
/sbin/multipath -F
/etc/init.d/multipathd restart
Now run multipath with the -ll option to see if you can see the 4 paths, and that the correct hardware-handler is being used. If everything is going well you should have something similar to the following.
multipath -ll
mpath0 (36001c23000c5a15e0000036747e24910) dm-3 DELL,MD3000i
[size=1.4T][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=200][active]
\_ 18:0:0:0 sdb 8:16 [active][ready]
\_ 19:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 17:0:0:0 sdd 8:48 [active][ghost]
\_ 20:0:0:0 sde 8:64 [active][ghost]
LVM configuration
To ensure that you don’t scan the ‘real’ devices and not the multipath devices, you need to change the filter in /etc/lvm/lvm.conf and rebuild the cache.
Find the filter line and change using your favourite editor.
####filter = [ "a/.*/" ] ###Original filter will be uncommented
filter = [ "a|/dev/sda|", "r|/dev/sd.*|", "r|/dev/dm-.*|", "a|/dev/mapper/mpath.*|", "r|/dev/.*|" ]
Now rebuild the cache.
rm /etc/lvm/cache/.cache
/sbin/vgscan
iSCSI mounting
Create a partition with fdisk
/sbin/fdisk /dev/sdb
The number of cylinders for this disk is set to 182038.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): m
n add a new partition
p print the partition table
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-182038, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-182038, default 182038):
Using default value 182038
Command (m for help): p
Disk /dev/sdb: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 182038 1462220203+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Create the new file system and mount it.
/usr/bin/nohup /sbin/mkfs.ext3 /dev/sdb1 &
/usr/bin/nohup: appending output to `nohup.out'
[1] 13610
[1]+ Done sudo /usr/bin/nohup /sbin/mkfs.ext3 /dev/sdb1
mkdir /mnt/iscsi
mount /dev/mapper/mpath0p1 /mnt/iscsi/
Verify you see the new partition
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
25G 13G 12G 51% /
/dev/sda1 99M 31M 64M 33% /boot
tmpfs 1014M 0 1014M 0% /dev/shm
/dev/mapper/VolGroup00-LogVol02
39G 6.3G 31G 17% /local
/dev/mapper/mpath0p1 1.4T 198M 1.3T 1% /mnt/iscsi
To mount the iSCSI disk on book ad it to /etc/fstab
/dev/mapper/mpath0p1 /mnt/iscsi ext3 _netdev 0 0
Gheek.net 