RTFM

[Read This Fine Material] from Joshua Hoblitt

How to migrate a GPFS 3.5 filesystem from mixed dataAndMetadata to using dedicated metadataOnly only disks

2013-06-07 by jhoblitt | 0 comments

(Or, How to migrate all GPFS filesystem metadata to SSDs)

IBM’s general advice [as of 2012] when setting up a new non-tired GPFS filesystem is to lean towards spreading data & metadata across all LUNs. See Data & Metadata in the GPFS wiki. Thus as if/when additional disks are added, the metadata performance will increase. I believe that this advice is fairly solid when using only rotational media. However, now that low cost / high write endurance MLC flashes devices are widely available (see my post on the Intel DC S3700 series), dramatic improvements in file system metadata operations are easy to achieve at low cost. I’ve now converted two GPFS clusters from using all mixed data & metadata disks to using a few SSDs as metadataOnly disks and converting the rest of the LUNs to being dataOnly. It is critical that before doing this you estimate how much metadata space your filesystem is currently using and will need in the immediate future. You will also need to monitor on going metadata usage of the filesystem.

This is the conversion procedure that I used.

First, check the configuration of the filesystem before planning any changes:
Continue Reading →

How to disable LSI MegaRAID SAS controller’s suspend boot on error “feature”

2013-06-06 by jhoblitt | 0 comments

LSI’s MegaRAID series of SAS RAID controllers will halt the system boot process when loading the cards option ROM (what LSI call’s the controller’s BIOS), if there are any “errors” detected. Manual intervention is needed to get the system boot process to pass the controller’s option ROM loading. Occasionally, this is a “good thing”, as more than once I’ve had a card’s self test has discover ECC memory problems. Much more often, this “feature” will hang the boot process due to the presence or “foreign configuration”. Ie, a disk that has MegaRAID configuration information on it that’s not part of active virtual device. For example, a hot spare moved from one system to another (without yet being reconfigured in the new system) can trigger this “feature”, halting the entire boot.

Example ECC error:


Continue Reading →

How to fix MegaRAID Storage Manager RPM preun scriplet errors

2013-06-06 by jhoblitt | 0 comments

I discovered this morning upgrading from MegaRAID Storage manager 12.08.03-03 -> 13.04.03-01 that the RPM for the old package wasn’t being removed. An example is below. Note that I’ve created local YUM repos for the LSI provided RPMs.

# yum update -y MegaCli MegaRAID_Storage_Manager
Loaded plugins: priorities, security, upgrade-helper
88 packages excluded due to repository priority protections
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package Lib_Utils2.noarch 0:1.00-04 will be updated
---> Package Lib_Utils2.noarch 0:1.00-05 will be an update
---> Package MegaCli.noarch 0:8.04.07-1 will be updated
---> Package MegaCli.noarch 0:8.07.07-1 will be an update
---> Package MegaRAID_Storage_Manager.noarch 0:12.08.03-03 will be updated
---> Package MegaRAID_Storage_Manager.noarch 0:13.04.03-01 will be an update
---> Package sas_ir_snmp.x86_64 0:12.08-0303 will be updated
---> Package sas_ir_snmp.x86_64 0:13.04-0301 will be an update
---> Package sas_snmp.x86_64 0:12.08-0303 will be updated
---> Package sas_snmp.x86_64 0:13.04-0301 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                        Arch         Version            Repository
                                                                           Size
================================================================================
Updating:
 Lib_Utils2                     noarch       1.00-05            lsi       1.1 M
 MegaCli                        noarch       8.07.07-1          lsi       1.5 M
 MegaRAID_Storage_Manager       noarch       13.04.03-01        lsi        65 M
 sas_ir_snmp                    x86_64       13.04-0301         lsi       936 k
 sas_snmp                       x86_64       13.04-0301         lsi       557 k

Transaction Summary
================================================================================
Upgrade       5 Package(s)

Total download size: 69 M
Downloading Packages:
(1/5): Lib_Utils2-1.00-05.noarch.rpm                     | 1.1 MB     00:00     
(2/5): MegaCli-8.07.07-1.noarch.rpm                      | 1.5 MB     00:00     
(3/5): MegaRAID_Storage_Manager-13.04.03-01.noarch.rpm   |  65 MB     00:00     
(4/5): sas_ir_snmp-13.04-0301.x86_64.rpm                 | 936 kB     00:00     
(5/5): sas_snmp-13.04-0301.x86_64.rpm                    | 557 kB     00:00     
--------------------------------------------------------------------------------
Total                                            64 MB/s |  69 MB     00:01     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : Lib_Utils2-1.00-05.noarch                                   1/10 
Installing....
  Updating   : sas_snmp-13.04-0301.x86_64                                  2/10 
Stopping LSI SNMP Agent:[  OK  ]
ln: creating symbolic link `/usr/lib64/libsas_objects.so.1': File exists
Starting snmpd
Stopping snmpd: [  OK  ]
Starting snmpd: [  OK  ]
Starting LSI SNMP Agent
Starting LSI SNMP Agent:LSI MegaRAID SNMP Agent Ver 3.18.0.2 (Jan 21st, 2013) Started
[  OK  ]
  Updating   : MegaCli-8.07.07-1.noarch                                    3/10 
  Updating   : sas_ir_snmp-13.04-0301.x86_64                               4/10 
Stopping LSI SNMP Agent:[  OK  ]
ln: creating symbolic link `/usr/lib64/libsas_ir_objects.so.1': File exists
Starting snmpd
Stopping snmpd: [  OK  ]
Starting snmpd: [  OK  ]
Starting LSI SNMP Agent
Starting LSI SNMP Agent:LSI MegaRAID SNMP Agent Ver 3.18.0.2 (Jan 21st, 2013) Started
[  OK  ]
  Updating   : MegaRAID_Storage_Manager-13.04.03-01.noarch                 5/10 
Installing....
/
/
/
Framework is already running.....
/
 
/
/
Error in PREUN scriptlet in rpm package MegaRAID_Storage_Manager
  Cleanup    : Lib_Utils2-1.00-04.noarch                                   6/10 
./popup: line 4: 26717 Killed                  ../jre/bin/java -classpath .:Popup.jar:../GUI.jar Popup.Communicator ajsgyqkj=71244 &>/dev/null
MegaRAID Storage Manager is running. Please shutdown the process and retry.
error: %preun(MegaRAID_Storage_Manager-12.08.03-03.noarch) scriptlet failed, exit status 1
  Cleanup    : MegaCli-8.04.07-1.noarch                                    7/10 
  Cleanup    : sas_snmp-12.08-0303.x86_64                                  8/10 
Stopping LSI SNMP Agent:[  OK  ]
  Cleanup    : sas_ir_snmp-12.08-0303.x86_64                               9/10 
Starting LSI SNMP Agent:LSI MegaRAID SNMP Agent Ver 3.18.0.2 (Jan 21st, 2013) Started
[  OK  ]
Stopping LSI SNMP Agent:[  OK  ]
Unregistering Service
Stopping snmpd: [  OK  ]
Starting snmpd: [  OK  ]
  Verifying  : sas_ir_snmp-13.04-0301.x86_64                               1/10 
  Verifying  : MegaRAID_Storage_Manager-13.04.03-01.noarch                 2/10 
  Verifying  : MegaCli-8.07.07-1.noarch                                    3/10 
  Verifying  : sas_snmp-13.04-0301.x86_64                                  4/10 
  Verifying  : Lib_Utils2-1.00-05.noarch                                   5/10 
MegaRAID_Storage_Manager-12.08.03-03.noarch was supposed to be removed but is not!
  Verifying  : MegaRAID_Storage_Manager-12.08.03-03.noarch                 6/10 
  Verifying  : Lib_Utils2-1.00-04.noarch                                   7/10 
  Verifying  : sas_snmp-12.08-0303.x86_64                                  8/10 
  Verifying  : MegaCli-8.04.07-1.noarch                                    9/10 
  Verifying  : sas_ir_snmp-12.08-0303.x86_64                              10/10 

Updated:
  Lib_Utils2.noarch 0:1.00-05                   MegaCli.noarch 0:8.07.07-1     
  MegaRAID_Storage_Manager.noarch 0:13.04.03-01 sas_ir_snmp.x86_64 0:13.04-0301
  sas_snmp.x86_64 0:13.04-0301                 

Failed:
  MegaRAID_Storage_Manager.noarch 0:12.08.03-03                                 

Complete!

And sure enough, the old RPM is still laying around.
Continue Reading →

How to [nicely] merge a git template repo into an existing project

2013-06-03 by jhoblitt | 1 Comment

A few months ago I created a simple template of a Puppet forge module to make clones of when starting a new module due to puppet module generate ... leaving much to be manually edited/created. This is very handy when starting from scratch but makes a mess of the commit history if just merging it into an existing puppet module repo. It turns out that git does have the ability to squash all commits from a branch during a merge.

Start by adding git@github.com:jhoblitt/puppet-module_skel.git as a remote for an existing repo.

$ git clone ssh://git@bitbucket.org/.../puppet-yum_base.git
Cloning into 'puppet-yum_base'...
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (4/4), done.
$ cd puppet-yum_base
$ git remote add skel git@github.com:jhoblitt/puppet-module_skel.git
$ git fetch skel

warning: no common commits
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 13 (delta 0), reused 13 (delta 0)
Unpacking objects: 100% (13/13), done.
From github.com:jhoblitt/puppet-module_skel
 * [new branch]      master     -> skel/master
$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/skel/master

Continue Reading →

Pacemaker 1.1.8 no longer includes the crm shell

2013-05-30 by jhoblitt | 0 comments

I just had a rather rude surprise upgrading from pacemaker 1.1.7 -> 1.1.8. The crm configuration shell has been removed from the distribution. One does not expect these sorts of changes to happen in a 1.1.z release.

v1.1.7

# rpm -qa | grep pacemaker
pacemaker-cli-1.1.7-6.el6.x86_64
pacemaker-libs-1.1.7-6.el6.x86_64
pacemaker-cluster-libs-1.1.7-6.el6.x86_64
pacemaker-1.1.7-6.el6.x86_64
# ls -la /usr/sbin/crm
-rwxr-xr-x 1 root root 1576 Jun 21  2012 /usr/sbin/crm

v1.1.8

# rpm -qa | grep pacemaker
pacemaker-cli-1.1.8-7.el6.x86_64
pacemaker-cluster-libs-1.1.8-7.el6.x86_64
pacemaker-1.1.8-7.el6.x86_64
pacemaker-libs-1.1.8-7.el6.x86_64
# ls -la /usr/sbin/crm
ls: cannot access /usr/sbin/crm: No such file or directory

Google finally turned up this thread on the pacemaker mailing list, stating this was an intentional change. More research found this note in Pacemaker’s README.markdown stating that this change was made at the request of the author and the CRM shell is now maintained else where. Except that it doesn’t work with pacemaker 1.1.8. It also appears that all of the official documentation now refers to pcs. At this point I’m thinking “Fine, I’ll just install pcs”.
Continue Reading →