RTFM

[Read This Fine Material] from Joshua Hoblitt

How to create a GPFS 3.5 cluster

2013-05-01 by jhoblitt | 0 comments

I recently realized that I never cleaned up and posted my notes about GPFS cluster creation. Here is a trivial example of a single node GPFS “cluster” with a single node in it.

Creating the cluster requires at least one qourum node. GPFS requires that any quorum node must be licensed as a GPFS server. Thus the smallest possible GPFS cluster is one GPFS server.

# echo "foo1:manager-quorum" > nodes.txt
[root@foo1 ~]# mmcrcluster -N nodes.txt -p foo1 -r /usr/bin/ssh -R /usr/bin/scp -C foo -U example.com -A
mmhost:getnameinfo: Cannot resolve 10.10.10.11; Name or service not known.
mmcrcluster: Incorrect node foo1 specified for command.
mmcrcluster: Command failed.  Examine previous error messages to determine cause.

It often bites me on a new cluster build that reverse DNS hasn’t yet been setup. This will cause cluster creation to fail, even with a single node.

# vim /etc/hosts
# mmcrcluster -N nodes.txt -p foo1 -r /usr/bin/ssh -R /usr/bin/scp -C foo -U example.com -A
Tue Apr 30 19:10:02 MST 2013: mmcrcluster: Processing node foo1
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.

We now need to “license” our node as a GPFS server.

# echo "foo1" > servers.txt
# mmchlicense server --accept -N servers.txt                

The following nodes will be designated as possessing GPFS server licenses:
    foo1
mmchlicense: Command successfully completed

And we should now have a complete cluster configuration.

# mmlscluster                

GPFS cluster information
========================
  GPFS cluster name:         foo.example.com
  GPFS cluster id:           12234795770926102905
  GPFS UID domain:           example.com
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:   foo1
  Secondary server:  (none)

 Node  Daemon node name  IP address     Admin node name  Designation
---------------------------------------------------------------------
   1   foo1              140.252.26.11  foo1             quorum-manager

Some cluster wide configuration directives can only be set while the the cluster is shutdown, now is a convient time to set those parameters since GPFS hasn’t been started yet.

# mmchconfig maxblocksize=4096K
Verifying GPFS is stopped on all nodes ...
mmchconfig: Command successfully completed
# mmchconfig maxMBpS=5000
mmchconfig: Command successfully completed
# mmchconfig autoload=yes # same as -A to mmcrcluster
mmchconfig: Command successfully completed
# mmchconfig pagepool=32G
mmchconfig: Command successfully completed
# mmchconfig nsdbufspace=50
mmchconfig: Command successfully completed
# mmchconfig maxStatCache=256K
mmchconfig: Command successfully completed
# mmchconfig maxFilesToCache=256K
mmchconfig: Command successfully completed
# mmchconfig seqDiscardThreshhold=1073741824
mmchconfig: Command successfully completed
# mmchconfig nfsPrefetchStrategy=1
mmchconfig: Command successfully completed
# mmlsconfig                 
Configuration data for cluster foo.example.com:
---------------------------------------------------
myNodeConfigNumber 1
clusterName foo.example.com
clusterId 12234795770926102905
autoload yes
uidDomain example.com
dmapiFileHandleSize 32
minReleaseLevel 3.5.0.3
maxblocksize 4096K
maxMBpS 5000
pagepool 32G 
nsdbufspace 50
maxStatCache 256K
nfsPrefetchStrategy 1
adminMode central

File systems in cluster foo.example.com:
--------------------------------------------
(none)

The final step is to startup GPFS.

# mmstartup -a
Tue Apr 30 19:29:36 MST 2013: mmstartup: Starting GPFS ...
# mmgetstate -a

 Node number  Node name        GPFS state
------------------------------------------
       1      foo1             active

Leave a Reply