RTFM

[Read This Fine Material] from Joshua Hoblitt

How to change the mount point of a GPFS filesystem

2012-04-05 by jhoblitt | 0 comments

This was a little non intuitive to me so I’ll document the procedure. What confused me is that, unlike most UNIXish filesystem, GPFS stores the mount path as a parameter in the filesystem metadata.

First use mmlsfs to find the block device name for the filesystem.

# mmlsfs all

File system attributes for /dev/foo:
=========================================
flag                value                    description
------------------- ------------------------ -----------------------------------
 -f                 131072                   Minimum fragment size in bytes
 -i                 512                      Inode size in bytes
 -I                 32768                    Indirect block size in bytes
 -m                 1                        Default number of metadata replicas
 -M                 2                        Maximum number of metadata replicas
 -r                 1                        Default number of data replicas
 -R                 2                        Maximum number of data replicas
 -j                 cluster                  Block allocation type
 -D                 nfs4                     File locking semantics in effect
 -k                 all                      ACL semantics in effect
 -n                 32                       Estimated number of nodes that will mount file system
 -B                 4194304                  Block size
 -Q                 none                     Quotas enforced
                    none                     Default quotas enabled
 --filesetdf        No                       Fileset df enabled?
 -V                 12.10 (3.4.0.7)          File system version
 --create-time      Wed Feb  8 18:34:14 2012 File system creation time
 -u                 Yes                      Support for large LUNs?
 -z                 No                       Is DMAPI enabled?
 -L                 4194304                  Logfile size
 -E                 Yes                      Exact mtime mount option
 -S                 No                       Suppress atime mount option
 -K                 whenpossible             Strict replica allocation option
 --fastea           Yes                      Fast external attributes enabled?
 --inode-limit      18300928                 Maximum number of inodes
 -P                 system                   Disk storage pools in file system
 -d                 nsd1;nsd2                Disks in file system
 -A                 yes                      Automatic mount option
 -o                 none                     Additional mount options
 -T                 /gpfs/bar                Default mount point
 --mount-priority   0                        Mount priority

Before attempting to modify the filesystem, you need to make sure that it unmounted on the cluster.

# mmumount /gpfs/bar -a
Wed Apr  4 17:50:07 MST 2012: mmumount: Unmounting file systems ...

The mount point can be modified with themmchfs command.

# mmchfs /dev/foo -T /net/bar
mmchfs: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

Then remount the filesystem on it’s new mountpoint.

# mmmount /net/bar -a
Wed Apr  4 17:51:57 MST 2012: mmmount: Mounting file systems ...

How to disable Intel(R) Embedded Server RAID Technology II

2012-04-02 by jhoblitt | 0 comments

Intel’s family of s5XX0 motherboards includes Embedded Server RAID Technology II. More or less this “technology” loads a BIOS option ROM that displays a status screen similar to typical hardware RAID controller. Unlike a hardware RAID controller, it doesn’t actual handle any redundancy functions. In a phrase, it’s yet another “fake” RAID controller. This particular implementation is frustrating in that it sets up a Linux dmraid device for you which you then have to partition. This doesn’t work with mdadm and it doesn’t work well scripted deployment.

IMHO – Fake RAID has no place in a sever environment as it does nothing for you that the tools included in virtually ever modern OS provide. It’s actually a step backwards in that it’s an extra setup step in provisioning that will be different not only for every hardware vendor but between different models. Once upon a time Fake RAID may have had a place the desktop but the basic BIOS ability to support a sequential list of boot devices makes it useless there as well. I do use software RAID1 on a great number of server and desktop systems and it’s nice to be able to use the same kickstart script across all classes of hardware.

On the S5520 series of boards, I’ve also encountered some strangeness with the AHCI ROM doing things like hanging the boot process infinitely if it doesn’t like partition table. After testing this under many different firmware revisions and after a failed attempt to get Intel to acknowledge it as a bug, I just set the SATA controller to ENHANCED mode and go on with life. Skipping this option ROM actually shorten the boot time on a motherboard that probably holds the crown for the longest x86 BIOS boot process (something to do with it’s dual UEFI/BIOSness?).

And here be dragons. If you change the SATA mode to ENHANCED or COMPATIBILITY after the RAID status screen vanishes from the boot process. However, if a fake raid set had already been created it does not go away. In disbelief, I’ve gone as far as physically pulling the drives from the system and zeroing them out with dd. The fake raid keeps coming back like it’s a T1000. The fix is to reenable the SATA SW RAID mode and clear out the entire configuration and, then change the SATA mode back again.

This is the menu option to completely dump the config:

This is what a system with two SATA disks and no Embedded RAID configuration looks like:

Scorpions fluoresce under UV light

2012-04-02 by jhoblitt | 0 comments

I’d read online that scorpions fluoresce under UV light but wasn’t unable to find any specification as to the wavelength and intensity required. All of the pictures of this phenomenon turned up by Google were of scorpions glowing in an atrium. Since most cheap UV LEDs seem to have a peak emission at 395nm, I assumed this was the wavelength being used. Wild scorpions, or at least the bark scorpions that venture into my living room, definitely will try to hide from visible light. I have no idea as to what spectrum scorpions see in but the “395nm” LEDs definitely produce a lot of light much redder than that. My thinking was that this “visible” spill over might cause a scorpion not trapped in an atrium to flee. I located an inexpensive low intensity 365nm flashlight online and purchased it.

This was a “proof of concept” I did a few weeks ago to see if the florescence would be bright enough to capture in a camera. The exposure was completely hand held (w/ IS) from a distance of ~2′ @ 1/10s, F/2.8, 55mm, ISO1600. The scorpion was on the order of 2-3″ in length and was partially hiding under the stuco in front of my garage door along the side of the house. The effect was bright enough that he was visible from ~10′ away with a well dark adapter mark I eye ball. If you look closely in the image you can see light from the scorpion reflecting off of the base of the garage door.

Re-adding a GPFS node to an existing cluster

2012-03-30 by jhoblitt | 2 Comments

IBM’s GPFS filesystem expects that all members of the GPFS cluster, be they clients or servers, have a complete and consistent state of the cluster. This causes some difficulty in re-adding a node to the cluster after it’s lost it’s configuration state for some reason. I frequently hit this problem when reprovision a compute node for some reason. The newly provisioned node will not allow to execute any GPFS commands as it knows that it’s not a member of a GPFS cluster. The other nodes of the cluster will also refuse to execute commands reguarding it because it’s state is inconsistent with their own perceived state of the cluster or “unknown”.

In this example, the node that has been reprovisioned from scratch is “dec06”.

As far as the rest of the cluster is conerned, that node is in a nonsensical state:

[root@dec01 ~]# mmgetstate -N dec06

 Node number  Node name        GPFS state 
------------------------------------------
       6      dec06            unknown

It can not be deleted from the cluster:

root@dec01 ~]# mmdelnode -N dec06
Verifying GPFS is stopped on all affected nodes ...
dec06.tuc.noao.edu:  mmremote: Unknown GPFS execution environment
mmdelnode: Command failed.  Examine previous error messages to determine cause.

Nor can GPFS be started on it:

[root@dec01 ~]# mmstartup -N dec06
Fri Mar 30 16:27:29 MST 2012: mmstartup: Starting GPFS ...
dec06.tuc.noao.edu:  mmremote: Unknown GPFS execution environment 
mmstartup: Command failed.  Examine previous error messages to determine cause.

The solution lives in the manpage for mmdelnode.

A node cannot be deleted if any of the following are true:
…
3. If the GPFS state is unknown and the node is reachable on the
network.

You cannot delete a node if both of the following are true:

– The node responds to a TCP/IP ping command from another node.

– The status of the node shows unknown when you use the mmget-
state command from another node in the cluster.

Thus, we just have to make the node unpingable for a few moments.

root@dec01 ~]# ssh dec06 reboot

Wait a few moments for the host to go down and then we can delete it from cluster.

[root@dec01 ~]# mmdelnode -N dec06
Verifying GPFS is stopped on all affected nodes ...
mmdelnode: Command successfully completed
mmdelnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

Wait another few moments for it come back online…

[root@dec01 ~]# ping dec06
PING dec06.tuc.noao.edu (140.252.27.26) 56(84) bytes of data.
64 bytes from dec06.tuc.noao.edu (140.252.27.26): icmp_seq=1 ttl=64 time=0.810 ms
^C
--- dec06.tuc.noao.edu ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 655ms
rtt min/avg/max/mdev = 0.810/0.810/0.810/0.000 ms

Then add it back into the cluster:

[root@dec01 ~]# mmaddnode -N dec06
Fri Mar 30 16:32:42 MST 2012: mmaddnode: Processing node dec06.tuc.noao.edu
mmaddnode: Command successfully completed
mmaddnode: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmaddnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@dec01 ~]# mmchlicense client -N dec06

The following nodes will be designated as possessing GPFS client licenses:
	dec06.tuc.noao.edu
Please confirm that you accept the terms of the GPFS client Licensing Agreement.
The full text can be found at www.ibm.com/software/sla
Enter "yes" or "no": yes
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

And the node is now back into a rational state:

[root@dec01 ~]# mmgetstate -N dec06

 Node number  Node name        GPFS state 
------------------------------------------
      18      dec06            down

DateTime::Format::ISO8601 0.08 released to CPAN

2012-02-12 by jhoblitt | 0 comments

This release is primarily to address an issue I noticed while evaluating this bug report: #52645: Can’t parse valid strings like “2009-12-10T09:00:00.00+0100”

From the Changes file:

0.08 Sat Feb 11 23:40:43 MST 2012
    - rt.cpan.org #52645 : UTC offsets must be in the same format
      (basic|extended) as the time as to which it is attached.

Available @: