Puppet tzdata Module version 1.0.1 released to Puppet Forge

2013-03-04 by jhoblitt | 0 comments

Changelog:

`v1.0.1`
========

Joshua Hoblitt  (7):
      * add a Rakefile with a lint target + setup for rspec with puppetlabs_spec_helper
      * change license from GPLv2 or later to Apache 2.0
      * enable travis-ci support
      * add a simple Changelog for the forge
      * change module version to v1.0.1
      * add .gitignore
      * add basic rspec tests

Puppet ganglia Module version 0.0.2 released to Puppet Forge

2013-03-03 by jhoblitt | 0 comments

Changelog:

`v0.0.2`
======

Changes since `v0.0.1`:

Joshua Hoblitt  (4):
      * cleanup code to pass puppet-lint and add the 'lint' rake target
      * fix gmond template namespace issues with puppet >= 3
      * enable travis-ci support
      * change license from GPLv2 or later to Apache 2.0

How to add disks to an existing GPFS 3.5.0 filesystem

2013-02-27 by jhoblitt | 0 comments

In a previous article titled How to add disks to an existing GPFS filesystem, I discussed adding additional disks/NSDs to a preexisting GPFS 3.4.0 filesystem. It turns out that the configuration file format used by mmcrnsd utility has completely changed between the 3.4.0.x and 3.5.0.0 releases. The old “DescFile” format used by has been replaced by the new “StanzaFile” format.

Here is a small portion of the mmcrnsd man page provided by the package gpfs.base-3.4.0-8.x86_64 that discusses the “DescFile” format expected by the command.

       Parameters
          -F DescFile
                Specifies  the  file  containing the list of disk descriptors,
                one per line. Disk descriptors have this format:

                DiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool
...

And here is the compared section of the mmcrnsd man page provided by the package gpfs.base-3.5.0-4.x86_64.

       Parameters
          -F StanzaFile
                Specifies the file containing the NSD stanzas for the disks to
                be created.  NSD stanzas have this format:

                %nsd: device=DiskName
                  nsd=NsdName
                  servers=ServerList
                  usage={dataOnly | metadataOnly | dataAndMetadata | descOnly}
                  failureGroup=FailureGroup
                  pool=StoragePool
...

Below is an example “StanzaFile” for a 3 server node GPFS cluster with 3 preexisting disks per node (no shared storage). I want to add an additional disk/NSD to all 3 systems and coincidentally the new block devices happens to be named /dev/sdf on all systems. This will increase the total number of GPFS disks in the cluster from 9 to 12.

Note that the “StanzaFile” format is not sensitive to blank lines between declarations so I’ve used a blank line to separate %nsd declarations for different nodes.

%nsd: device=sdc
  nsd=foo_nsd1
  servers=foo
  usage=dataAndMetadata
  failureGroup=1
  pool=system
%nsd: device=sdd
  nsd=foo_nsd2
  servers=foo
  usage=dataAndMetadata
  failureGroup=1
  pool=system
%nsd: device=sde
  nsd=foo_nsd3
  servers=foo
  usage=dataAndMetadata
  failureGroup=1
  pool=system
%nsd: device=sdf
  nsd=foo_nsd4
  servers=foo
  usage=dataAndMetadata
  failureGroup=1
  pool=system

%nsd: device=sdc
  nsd=foo2_nsd1
  servers=foo2
  usage=dataAndMetadata
  failureGroup=2
  pool=system
%nsd: device=sdd
  nsd=foo2_nsd2
  servers=foo2
  usage=dataAndMetadata
  failureGroup=2
  pool=system
%nsd: device=sde
  nsd=foo2_nsd3
  servers=foo2
  usage=dataAndMetadata
  failureGroup=2
  pool=system
%nsd: device=sdf
  nsd=foo2_nsd4
  servers=foo2
  usage=dataAndMetadata
  failureGroup=2
  pool=system

%nsd: device=sdc
  nsd=foo3_nsd1
  servers=foo3
  usage=dataAndMetadata
  failureGroup=3
  pool=system
%nsd: device=sdd
  nsd=foo3_nsd2
  servers=foo3
  usage=dataAndMetadata
  failureGroup=3
  pool=system
%nsd: device=sde
  nsd=foo3_nsd3
  servers=foo3
  usage=dataAndMetadata
  failureGroup=3
  pool=system
%nsd: device=sdf
  nsd=foo3_nsd4
  servers=foo3
  usage=dataAndMetadata
  failureGroup=3
  pool=system

This is the present state of the cluster.

[root@foo1 ~]# mmlsnsd -X

 Disk name    NSD volume ID      Device         Devtype  Node name                Remarks          
---------------------------------------------------------------------------------------------------
 foo1_nsd1 8CFC1C05507CC5D0   /dev/sda       generic  foo1.example.org     server node
 foo1_nsd2 8CFC1C05507CC5D1   /dev/sdb       generic  foo1.example.org     server node
 foo1_nsd3 8CFC1C05507CC5D2   /dev/sdc       generic  foo1.example.org     server node
 foo2_nsd1 8CFC1C08507CC5D3   /dev/sdc       generic  foo2.example.org     server node
 foo2_nsd2 8CFC1C08507CC5D6   /dev/sdd       generic  foo2.example.org     server node
 foo2_nsd3 8CFC1C08507CC5D8   /dev/sde       generic  foo2.example.org     server node
 foo3_nsd1 8CFC1C0B507CC600   /dev/sda       generic  foo3.example.org     server node
 foo3_nsd2 8CFC1C0B507CC602   /dev/sdb       generic  foo3.example.org     server node
 foo3_nsd3 8CFC1C0B507CC605   /dev/sdc       generic  foo3.example.org     server node

And now we pass the “StanzaFile” that includes the new and old NSD definitions to mmcrnsd.

[root@foo1 ~]# mmcrnsd -F stanzafile.txt
mmcrnsd: Processing disk sdc
mmcrnsd: Disk name foo1_nsd1 is already registered for use by GPFS.
mmcrnsd: Processing disk sdd
mmcrnsd: Disk name foo1_nsd2 is already registered for use by GPFS.
mmcrnsd: Processing disk sde
mmcrnsd: Disk name foo1_nsd3 is already registered for use by GPFS.
mmcrnsd: Processing disk sdf
mmcrnsd: Processing disk sdc
mmcrnsd: Disk name foo2_nsd1 is already registered for use by GPFS.
mmcrnsd: Processing disk sdd
mmcrnsd: Disk name foo2_nsd2 is already registered for use by GPFS.
mmcrnsd: Processing disk sde
mmcrnsd: Disk name foo2_nsd3 is already registered for use by GPFS.
mmcrnsd: Processing disk sdf
mmcrnsd: Processing disk sdc
mmcrnsd: Disk name foo3_nsd1 is already registered for use by GPFS.
mmcrnsd: Processing disk sdd
mmcrnsd: Disk name foo3_nsd2 is already registered for use by GPFS.
mmcrnsd: Processing disk sde
mmcrnsd: Disk name foo3_nsd3 is already registered for use by GPFS.
mmcrnsd: Processing disk sdf
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

This is what stanzafile.txt looks like after having been processed by mmcrnsd. Note that as with the “DescFile” format from GPFS 3.4.0, mmcrnsd has commented out all of the preexisting NSDs.

# %nsd: device=sdc
#   nsd=foo1_nsd1
#   servers=foo1
#   usage=dataAndMetadata
#   failureGroup=1
#   pool=system
# %nsd: device=sdd
#   nsd=foo1_nsd2
#   servers=foo1
#   usage=dataAndMetadata
#   failureGroup=1
#   pool=system
# %nsd: device=sde
#   nsd=foo1_nsd3
#   servers=foo1
#   usage=dataAndMetadata
#   failureGroup=1
#   pool=system
%nsd: device=sdf
  nsd=foo1_nsd4
  servers=foo1
  usage=dataAndMetadata
  failureGroup=1
  pool=system

# %nsd: device=sdc
#   nsd=foo2_nsd1
#   servers=foo2
#   usage=dataAndMetadata
#   failureGroup=2
#   pool=system
# %nsd: device=sdd
#   nsd=foo2_nsd2
#   servers=foo2
#   usage=dataAndMetadata
#   failureGroup=2
#   pool=system
# %nsd: device=sde
#   nsd=foo2_nsd3
#   servers=foo2
#   usage=dataAndMetadata
#   failureGroup=2
#   pool=system
%nsd: device=sdf
  nsd=foo2_nsd4
  servers=foo2
  usage=dataAndMetadata
  failureGroup=2
  pool=system

# %nsd: device=sdc
#   nsd=foo3_nsd1
#   servers=foo3
#   usage=dataAndMetadata
#   failureGroup=3
#   pool=system
# %nsd: device=sdd
#   nsd=foo3_nsd2
#   servers=foo3
#   usage=dataAndMetadata
#   failureGroup=3
#   pool=system
# %nsd: device=sde
#   nsd=foo3_nsd3
#   servers=foo3
#   usage=dataAndMetadata
#   failureGroup=3
#   pool=system
%nsd: device=sdf
  nsd=foo3_nsd4
  servers=foo3
  usage=dataAndMetadata
  failureGroup=3
  pool=system

Here is a listing of defined NSDs in the cluster after mmcrnsd has completed. Note the new X_nsd4 /dev/sdf NSDs per node.

[root@foo1 ~]# mmlsnsd -X

 Disk name    NSD volume ID      Device         Devtype  Node name                Remarks
---------------------------------------------------------------------------------------------------
 foo1_nsd1 8CFC1C05507CC5D0   /dev/sda       generic  foo1.example.org     server node
 foo1_nsd2 8CFC1C05507CC5D1   /dev/sdb       generic  foo1.example.org     server node
 foo1_nsd3 8CFC1C05507CC5D2   /dev/sdc       generic  foo1.example.org     server node
 foo1_nsd4 8CFC1C055122CB38   /dev/sdf       generic  foo1.example.org     server node
 foo2_nsd1 8CFC1C08507CC5D3   /dev/sdc       generic  foo2.example.org     server node
 foo2_nsd2 8CFC1C08507CC5D6   /dev/sdd       generic  foo2.example.org     server node
 foo2_nsd3 8CFC1C08507CC5D8   /dev/sde       generic  foo2.example.org     server node
 foo2_nsd4 8CFC1C085122CB3A   /dev/sdf       generic  foo2.example.org     server node
 foo3_nsd1 8CFC1C0B507CC600   /dev/sda       generic  foo3.example.org     server node
 foo3_nsd2 8CFC1C0B507CC602   /dev/sdb       generic  foo3.example.org     server node
 foo3_nsd3 8CFC1C0B507CC605   /dev/sdc       generic  foo3.example.org     server node
 foo3_nsd4 8CFC1C0B5122CB3C   /dev/sdf       generic  foo3.example.org     server node

As with GPFS 3.4.0.x, the new disks are not yet part of an existing filesystem.

[root@foo1 ~]# mmlsdisk foo1
disk         driver   sector failure holds    holds                            storage
name         type       size   group metadata data  status        availability pool
------------ -------- ------ ------- -------- ----- ------------- ------------ ------------
foo1_nsd1 nsd         512       1 Yes      Yes   ready         up           system          
foo1_nsd2 nsd         512       1 Yes      Yes   ready         up           system          
foo1_nsd3 nsd         512       1 Yes      Yes   ready         up           system          
foo2_nsd1 nsd         512       2 Yes      Yes   ready         up           system          
foo2_nsd2 nsd         512       2 Yes      Yes   ready         up           system          
foo2_nsd3 nsd         512       2 Yes      Yes   ready         up           system          
foo3_nsd1 nsd         512       3 Yes      Yes   ready         up           system          
foo3_nsd2 nsd         512       3 Yes      Yes   ready         up           system          
foo3_nsd3 nsd         512       3 Yes      Yes   ready         up           system

Now we’re going to add those 3 new disks to a prexisting filesystem with the mmadddisk. This works exactly the same as with GPFS 3.4.0.x accept mmadddisk expects a processed “StanzaFile” instead of a “DescFile”.

[root@foo1 ~]# mmadddisk foo1 -F ./stanzafile.txt

The following disks of mss1 will be formatted on node foo2.example.org:
    foo1_nsd4: size 31251759104 KB 
    foo2_nsd4: size 31251759104 KB 
    foo3_nsd4: size 31251759104 KB 
Extending Allocation Map
Checking Allocation Map for storage pool system
  58 % complete on Mon Feb 18 18:25:09 2013
 100 % complete on Mon Feb 18 18:25:13 2013
Completed adding disks to file system mss1.
mmadddisk: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

Now the 3 additional disks should be visible as part of the foo1 filesystem. Note the 3 fooX_nsd4 listings at the bottom of the output from mmlsdisk.

[root@foo1 work]# mmlsdisk mss1
disk         driver   sector failure holds    holds                            storage
name         type       size   group metadata data  status        availability pool
------------ -------- ------ ------- -------- ----- ------------- ------------ ------------
foo1_nsd1 nsd         512       1 Yes      Yes   ready         up           system   
foo1_nsd2 nsd         512       1 Yes      Yes   ready         up           system   
foo1_nsd3 nsd         512       1 Yes      Yes   ready         up           system   
foo2_nsd1 nsd         512       2 Yes      Yes   ready         up           system   
foo2_nsd2 nsd         512       2 Yes      Yes   ready         up           system   
foo2_nsd3 nsd         512       2 Yes      Yes   ready         up           system   
foo3_nsd1 nsd         512       3 Yes      Yes   ready         up           system   
foo3_nsd2 nsd         512       3 Yes      Yes   ready         up           system   
foo3_nsd3 nsd         512       3 Yes      Yes   ready         up           system   
foo1_nsd4 nsd         512       1 Yes      Yes   ready         up           system   
foo2_nsd4 nsd         512       2 Yes      Yes   ready         up           system   
foo3_nsd4 nsd         512       3 Yes      Yes   ready         up           system

The foo1 filesystem now consists of 12 disks of equal size. However, 9 of the disks are at less than or equal to 20% free capacity while 3 disks are almost completely unused.


[root@foo1 ~]# mmdf foo1
disk                disk size  failure holds    holds              free KB             free KB
name                    in KB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 233 TB)
foo1_nsd1      31251759104        1 Yes      Yes      5145806848 ( 16%)     236481600 ( 1%)
foo1_nsd2      31251759104        1 Yes      Yes      5142163456 ( 16%)     251675200 ( 1%)
foo1_nsd3      31251759104        1 Yes      Yes      5240137728 ( 17%)     131968576 ( 0%)
foo1_nsd4      31251759104        1 Yes      Yes     31249270784 (100%)         20928 ( 0%)
foo2_nsd2      31251759104        2 Yes      Yes      5144201216 ( 16%)     176784256 ( 1%)
foo2_nsd3      31251759104        2 Yes      Yes      5979107328 ( 19%)     114685120 ( 0%)
foo2_nsd4      31251759104        2 Yes      Yes     31249854464 (100%)         17856 ( 0%)
foo2_nsd1      31251759104        2 Yes      Yes      5144209408 ( 16%)     223169664 ( 1%)
foo3_nsd3      31251759104        3 Yes      Yes      6262112256 ( 20%)     112285056 ( 0%)
foo3_nsd2      31251759104        3 Yes      Yes      5143769088 ( 16%)     143519552 ( 0%)
foo3_nsd1      31251759104        3 Yes      Yes      5142896640 ( 16%)     245539392 ( 1%)
foo3_nsd4      31251759104        3 Yes      Yes     31249786880 (100%)         21120 ( 0%)
                -------------                         -------------------- -------------------
(pool total)     375021109248                          142093316096 ( 38%)    1636168320 ( 0%)

                =============                         ==================== ===================
(total)          375021109248                          142093316096 ( 38%)    1636168320 ( 0%)

Inode Information
-----------------
Number of used inodes:        51937690
Number of free inodes:        18177638
Number of allocated inodes:   70115328
Maximum number of inodes:    146800640

Here’s the exact same output rounded to the nearest TiB for bragging rights. 🙂

[root@foo1 ~]# mmdf foo1 --block-size 1T
disk                disk size  failure holds    holds              free TB             free TB
name                    in TB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 233 TB)
foo1_nsd1               30        1 Yes      Yes               5 ( 16%)             1 ( 1%)
foo1_nsd2               30        1 Yes      Yes               5 ( 16%)             1 ( 1%)
foo1_nsd3               30        1 Yes      Yes               5 ( 17%)             1 ( 0%)
foo1_nsd4               30        1 Yes      Yes              30 (100%)             1 ( 0%)
foo2_nsd2               30        2 Yes      Yes               5 ( 16%)             1 ( 1%)
foo2_nsd3               30        2 Yes      Yes               6 ( 19%)             1 ( 0%)
foo2_nsd4               30        2 Yes      Yes              30 (100%)             1 ( 0%)
foo2_nsd1               30        2 Yes      Yes               5 ( 16%)             1 ( 1%)
foo3_nsd3               30        3 Yes      Yes               6 ( 20%)             1 ( 0%)
foo3_nsd2               30        3 Yes      Yes               5 ( 16%)             1 ( 0%)
foo3_nsd1               30        3 Yes      Yes               5 ( 16%)             1 ( 1%)
foo3_nsd4               30        3 Yes      Yes              30 (100%)             1 ( 0%)
                -------------                         -------------------- -------------------
(pool total)              350                                   133 ( 38%)             2 ( 0%)

                =============                         ==================== ===================
(total)                   350                                   133 ( 38%)             2 ( 0%)

Inode Information
-----------------
Number of used inodes:        51937685
Number of free inodes:        18177643
Number of allocated inodes:   70115328
Maximum number of inodes:    146800640

It’s not strictly necessary at this point but it may be desirable to restripe the filesystem to even the load across LUNs and possibly increase the performance of reading existing large files (by virtue of them becoming striped across additional LUNs). I like to limit the nodes participating in the restriping to just those acting as NSD servers. Eg.

time mmrestripefs mss1 -b -N foo1,foo2,foo3

how to remove all Ruby Gems except those installed by system packages

2013-02-27 by jhoblitt | 0 comments

I’m sure it’s never happened to you but for me, occasionally, bundler or even the gem command pulls in a different version of a Gem that was installed by a standard system package. Even more rarely, this breaks some other software that was installed on the system.

This one-liner will remove ALL gems installed on a system.

gem search --local --no-version | tail -n +3 | xargs gem uninstall --all

The tail -n +3 command is needed to remove the \n*** LOCAL GEMS ***\n header that gem spits out in search mode.

That one-linger does not affect the platform’s package database but does nuke all of the gem associated files that may have been installed by a regular package. Since the system package database was untouched, now it’s just a matter of finding out which packages installed a gem reinstalling them. On EL systems, the names of these package are highly consistent and will begin with rubygem-.

rpm -qa | grep '^rubygem-' | xargs yum reinstall -y

At this point, in theory, we have scrubbed the system clean of all gems that were installed and reinstalled any gem providing RPM that’s consistent with the platforms naming guidelines. However, it is possible that a poorly named package was missed so do warrant some caution in using this procedure.

Here are the two one-liners together for an easy cut’n’paste.

gem search --local --no-version | tail -n +3 | xargs gem uninstall --all
rpm -qa | grep '^rubygem-' | xargs yum reinstall -y

mount/umount -a flag combined with -t

2013-02-25 by jhoblitt | 0 comments

I’m not sure why it had never occurred to me before this morning but it’s trivial to umount just NFS mounted filesystems by combing umount‘s -t and -a flags. This is particularly useful for fixing hung NFS mounts.

Attempt to nicely unmount all NFS mounts: