Gheek.net

July 12, 2011

A single tool to compare text, binary, picture, etc files.

Filed under: Uncategorized — lancevermilion @ 1:29 pm

Just a thought for those of you that need to use a text, binary compare tool here and there.

Today I had to waste some time and figure out what text, binary, picture, etc compare tool was the best. I Compared a handful of tools I used in the past and some I haven’t used. gVIM Diff, WinMerge, Beyond Compare, Perforce, and DiffMerge. I decided that a few of my needs was to do a diff on the fly without creating a file, show special characters, syntax highlighting, merging, line numbering, pic split view, search within both files, SVN integration, etc. I ended up going with Beyond Comapre since we have a corp license and kep WinMerge because it was a very close second and has integration to TortoiseSVN which I use as well.

gVIM – http://www.vim.org/download.php
WinMerge – http://winmerge.org/downloads/
BeyondComapre – http://www.scootersoftware.com/download.php
Perforce (P4V) – http://www.perforce.com/perforce/downloads/index.html
DiffMerge – http://www.sourcegear.com/diffmerge/downloads.php
TortoiseSVN – http://tortoisesvn.net/downloads.html

Advertisements

November 18, 2008

Multipath.conf/iscsi.conf install and howto

Filed under: Uncategorized — lancevermilion @ 10:45 am

These install/howto instructions are for CentOS 5.x and Dell MD3000i. You can easily modify these slightly for your specific iSCSI disk.

iSCSI/Multipath/LVM

iSCSI/Multipath Installation

Install iSCSI initiator utilities

yum install iscsi-initiator-utils

Install multipath device mapper

yum install device-mapper-multipath

Configure multipath, iscsi, and iscsid to start on startup. Check first to make sure they are not already set to start on startup. If they are set to start you will see something like below, if they are not you will not see anything output listed as “on”.

/sbin/chkconfig --list | egrep -i "iscsi|iscsid|multipathd"
iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

To configure multipathd to start on startup do the following. iSCSI(d)already have this defined in their init script.

/sbin/chkconfig multipathd on

iSCSI Configuration

Edit /etc/iscsi/iscsid.conf and make sure it is identical to this config.
**I need to verify this iscsid.conf but for now this is what I have logged as a working configs from years back.

#
# Open-iSCSI default configuration.
# Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf
#
# Note: To set any of these values for a specific node/session run
# the iscsiadm --mode node --op command for the value. See the README
# and man page for iscsiadm for details on the --op command.
#

################
# iSNS settings
################
# Address of iSNS server
#isns.address = 192.168.0.1
#isns.port = 3205

#############################
# NIC/HBA and driver settings
#############################
# open-iscsi can create a session and bind it to a NIC/HBA.
# To set this up see the example iface config file.

#*****************
# Startup settings
#*****************

# To request that the iscsi initd scripts startup a session set to "automatic".
# node.startup = automatic
#
# To manually startup the session set to "manual". The default is automatic.
node.startup = automatic

# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = username
#node.session.auth.password = password

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = username_in
#node.session.auth.password_in = password_in

# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
#discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
#discovery.sendtargets.auth.username = username
#discovery.sendtargets.auth.password = password

# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = username_in
#discovery.sendtargets.auth.password_in = password_in

# ********
# Timeouts
# ********
#
# See the iSCSI REAME's Advanced Configuration section for tips
# on setting timeouts when using multipath or doing root over iSCSI.
#
# To specify the length of time to wait for session re-establishment
# before failing SCSI commands back to the application when running
# the Linux SCSI Layer error handler, edit the line.
# The value is in seconds and the default is 120 seconds.
node.session.timeo.replacement_timeout = 120

# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 15

# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 15

# Time interval to wait for on connection before sending a ping.
node.conn[0].timeo.noop_out_interval = 5

# To specify the time to wait for a Nop-out response before failing
# the connection, edit this line. Failing the connection will
# cause IO to be failed back to the SCSI layer. If using dm-multipath
# this will cause the IO to be failed to the multipath layer.
node.conn[0].timeo.noop_out_timeout = 5

#******
# Retry
#******

# To speficy the number of times iscsiadm should retry a login
# to the target when we first login, modify the following line.
# The default is 4. Valid values are any integer value. This only
# affects the initial login. Setting it to a high value can slow
# down the iscsi service startup. Setting it to a low value can
# cause a session to not get logged into, if there are distuptions
# during startup or if the network is not ready at that time.
node.session.initial_login_retry_max = 4

################################
# session and device queue depth
################################

# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 128

# To control the device's queue depth set node.session.queue_depth
# to a value between 1 and 128. The default is 32.
node.session.queue_depth = 32

#***************
# iSCSI settings
#***************

# To enable R2T flow control (i.e., the initiator must wait for an R2T
# command before sending any data), uncomment the following line:
#
#node.session.iscsi.InitialR2T = Yes
#
# To disable R2T flow control (i.e., the initiator has an implied
# initial R2T of "FirstBurstLength" at offset 0), uncomment the following line:
#
# The defaults is No.
node.session.iscsi.InitialR2T = No

#
# To disable immediate data (i.e., the initiator does not send
# unsolicited data with the iSCSI command PDU), uncomment the following line:
#
#node.session.iscsi.ImmediateData = No
#
# To enable immediate data (i.e., the initiator sends unsolicited data
# with the iSCSI command packet), uncomment the following line:
#
# The default is Yes
node.session.iscsi.ImmediateData = Yes

# To specify the maximum number of unsolicited data bytes the initiator
# can send in an iSCSI PDU to a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144
node.session.iscsi.FirstBurstLength = 262144

# To specify the maximum SCSI payload that the initiator will negotiate
# with the target for, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the defauls it 16776192
node.session.iscsi.MaxBurstLength = 16776192

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 131072
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target during a discovery session, edit the
# following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 32768
#
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768

# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of enabling the checking, uncommen
# the following lines (Data digests are not supported and on ppc/ppc64
# both header and data digests are not supported.):
#node.conn[0].iscsi.HeaderDigest = CRC32C,None
#
# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of disabling the checking,
# uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = None,CRC32C
#
# To enable CRC32C digest checking for the header and/or data part of
# iSCSI PDUs, uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C
#
# To disable digest checking for the header and/or data part of
# iSCSI PDUs, uncomment the following lines:
#node.conn[0].iscsi.HeaderDigest = None
#
# The default is to never use DataDigests and to allow the target to control
# the setting of the HeaderDigest checking with the initiator requesting
# a preference of disabling the checking.

#***************
# iSCSI Multipath settings
#***************
# Specify the use of having multiple network paths to the same target.
# Valid values are
# no - there is no multipathing. Establish at most one session to the
# target and use all portals for failover, if failover is not disabled.
# portal - establish a session to the target using each of the n/w portals to
# the target, and bind each session to a different host SCSI address.
# No portal failover is possible since each session has only one portal.
# portalgroup - establish a session to one portal from each portal group that
# reaches the target and bind each session to a different host SCSI
# address. If a portal group has more than one portal, portal failover
# can occur between that group's portals if failover is not disabled.
Multipath = portalgroup

Edit /etc/iscsi/initiatorname.iscsi and add the alias name you created (in the MD3000i for this host) to this config.

InitiatorAlias=iqn.SERVER_HOSTNAME

Discover the iSCSI target.

iscsiadm -m discovery -t sendtargets -p 192.168.130.101

192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f
192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f

Login to the iSCSI target.

iscsiadm -m node -l

Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.101,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.102,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.101,3260]
Login session [iface: default, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.102,3260]

Verify you are logged in to the iSCSI target and have a session

iscsiadm -m session

tcp: [5] 192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [6] 192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [7] 192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5
tcp: [8] 192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.6001c23000c5a15e00000000471474b5

Logout of the iSCSI target (if you ever need to).

iscsiadm -m node -u

Logout session [sid: 9, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.101,3260]
Logout session [sid: 10, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.102,3260]
Logout session [sid: 11, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.131.101,3260]
Logout session [sid: 12, target: iqn.1984-05.com.dell:powervault.6001c23000c66e31000000004715e09f, portal: 192.168.130.102,3260]

Verify that the iSCSI target has you as a host and has disks mapped to you.
You will want to ignore any Disk the doesn’t contain a valid partition table
You can also check /var/log/dmesg to see similar but more verbose info, but is not covered here.

fdisk -l

Disk /dev/sda: 72.7 GB, 72746008576 bytes
255 heads, 63 sectors/track, 8844 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 8844 70935007+ 8e Linux LVM

Disk /dev/sdb: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdf: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes

Disk /dev/sdf doesn't contain a valid partition table

Disk /dev/sdg: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes

Disk /dev/sdg doesn't contain a valid partition table

Disk /dev/sdh: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes

Disk /dev/sdh doesn't contain a valid partition table

Disk /dev/sdi: 20 MB, 20971520 bytes
1 heads, 40 sectors/track, 1024 cylinders
Units = cylinders of 40 * 512 = 20480 bytes

Disk /dev/sdi doesn't contain a valid partition table

Disk /dev/dm-3: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-3 doesn't contain a valid partition table

Multipath Configuration

Now that we have the modules compiled and installed, we need to setup a configuration file so that the multipath modules/daemon will choose the correct devices.

Edit /etc/multipath.conf‘ by commenting out everything that is not already commented out. Then insert the following

# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated

defaults {
        udev_dir                /dev
        user_friendly_names     yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^sda"
        devnode "^sda[0-9]"
        device {
                vendor DELL
                product "PERC|Universal|Virtual"
        }
}
devices {
        device {
                vendor                  DELL
                product                 MD3000i
                hardware_handler        "1 rdac"
                path_checker            rdac
                path_grouping_policy    group_by_prio
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
                failback                immediate
                getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        }
}
multipaths {
        mulitpath {
                device {
                        vendor DELL
                        product MD3000i
                }
        }
}

Multipath Discovery

Now we have a configuration and modules, lets clear multipath database and re-establish the paths.

To flush the database use -F and the restart the deamon.

/sbin/multipath -F
/etc/init.d/multipathd restart
Now run multipath with the -ll option to see if you can see the 4 paths, and that the correct hardware-handler is being used. If everything is going well you should have something similar to the following.

multipath -ll
mpath0 (36001c23000c5a15e0000036747e24910) dm-3 DELL,MD3000i
[size=1.4T][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=200][active]
\_ 18:0:0:0 sdb 8:16 [active][ready]
\_ 19:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 17:0:0:0 sdd 8:48 [active][ghost]
\_ 20:0:0:0 sde 8:64 [active][ghost]

LVM configuration

To ensure that you don’t scan the ‘real’ devices and not the multipath devices, you need to change the filter in /etc/lvm/lvm.conf and rebuild the cache.
Find the filter line and change using your favourite editor.

####filter = [ "a/.*/" ] ###Original filter will be uncommented
filter = [ "a|/dev/sda|", "r|/dev/sd.*|", "r|/dev/dm-.*|", "a|/dev/mapper/mpath.*|", "r|/dev/.*|" ]

Now rebuild the cache.

rm /etc/lvm/cache/.cache
/sbin/vgscan

iSCSI mounting

Create a partition with fdisk

/sbin/fdisk /dev/sdb

The number of cylinders for this disk is set to 182038.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m

n add a new partition
p print the partition table

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-182038, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-182038, default 182038):
Using default value 182038

Command (m for help): p

Disk /dev/sdb: 1497.3 GB, 1497314099200 bytes
255 heads, 63 sectors/track, 182038 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 182038 1462220203+ 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Create the new file system and mount it.

/usr/bin/nohup /sbin/mkfs.ext3 /dev/sdb1 &
/usr/bin/nohup: appending output to `nohup.out'
[1] 13610
[1]+ Done sudo /usr/bin/nohup /sbin/mkfs.ext3 /dev/sdb1

mkdir /mnt/iscsi
mount /dev/mapper/mpath0p1 /mnt/iscsi/

Verify you see the new partition

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
25G 13G 12G 51% /
/dev/sda1 99M 31M 64M 33% /boot
tmpfs 1014M 0 1014M 0% /dev/shm
/dev/mapper/VolGroup00-LogVol02
39G 6.3G 31G 17% /local
/dev/mapper/mpath0p1 1.4T 198M 1.3T 1% /mnt/iscsi

To mount the iSCSI disk on book ad it to /etc/fstab

/dev/mapper/mpath0p1 /mnt/iscsi ext3 _netdev 0 0

October 14, 2008

MySQL add an index to a table.

Filed under: Uncategorized — lancevermilion @ 11:06 am

This is how to add a index to a table. Adding this single index for syslog data added about a index (idx) file of about 12% of the size of the sys_log table data size.
alter table sys_log add index (timestamp);

Referenced from: mysqlperformanceblog.com
Finding largest tables on MySQL instance is no brainier in MySQL 5.0+ thanks to Information Schema but I still wanted to post little query I use for the purpose so I can easily find it later, plus it is quite handy in a way it presents information:
SQL:
mysql> SELECT concat(table_schema,'.',table_name),concat(round(table_rows/1000000,2),'M') rows,concat(round(data_length/(1024*1024*1024),2),'G') DATA,concat(round(index_length/(1024*1024*1024),2),'G') idx,concat(round((data_length+index_length)/(1024*1024*1024),2),'G') total_size,round(index_length/data_length,2) idxfrac FROM information_schema.TABLES ORDER BY data_length+index_length DESC LIMIT 10;
+-------------------------------------+--------+--------+--------+------------+---------+
| concat(table_schema,'.',table_name) | rows | DATA | idx | total_size | idxfrac |
+-------------------------------------+--------+--------+--------+------------+---------+
| art87.link_out87 | 37.25M | 14.83G | 14.17G | 29.00G | 0.96 |
| art87.article87 | 12.67M | 15.83G | 4.79G | 20.62G | 0.30 |
| art116.article116 | 10.49M | 12.52G | 3.65G | 16.18G | 0.29 |
| art84.article84 | 10.10M | 10.11G | 3.59G | 13.70G | 0.35 |
| art104.link_out104 | 23.66M | 6.63G | 6.55G | 13.18G | 0.99 |
| art118.article118 | 7.06M | 10.49G | 2.68G | 13.17G | 0.26 |
| art106.article106 | 9.86M | 10.19G | 2.76G | 12.95G | 0.27 |
| art85.article85 | 6.20M | 9.82G | 2.51G | 12.33G | 0.26 |
| art91.article91 | 8.66M | 9.17G | 2.66G | 11.83G | 0.29 |
| art94.article94 | 5.21M | 10.10G | 1.69G | 11.79G | 0.17 |
+-------------------------------------+--------+--------+--------+------------+---------+
10 rows IN SET (2 min 29.19 sec)

Other good mysql links from mysqlperformaceblog.com
http://www.mysqlperformanceblog.com/2008/03/17/researching-your-mysql-table-sizes/

February 15, 2008

Red Team Green Team

Filed under: Uncategorized — lancevermilion @ 3:24 pm

If you work for a large company it will remind you of exactly how they do things. LOL. Worth taking a moment to view this one.

Create a free website or blog at WordPress.com.