RTFM

[Read This Fine Material] from Joshua Hoblitt

How to add a new client node to an existing GPFS 3.5.0 cluster

| 0 comments

Single node at a time.

Commands need to be run from a node with GPFS administrative privileges (the ability to ssh into client nodes as root).

# mmaddnode -N foo19
Fri Nov 21 15:01:17 MST 2014: mmaddnode: Processing node foo19.example.org
mmaddnode: Command successfully completed
mmaddnode: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmaddnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
# mmchlicense client --accept -N foo19

The following nodes will be designated as possessing GPFS client licenses:
	foo19.example.org
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
# mmlsnode -N foo19
foo19.example.org
# mmgetstate -N foo19

 Node number  Node name        GPFS state 
------------------------------------------
      26      foo19            down
# mmgetstate -N foo19

 Node number  Node name        GPFS state 
------------------------------------------
      26      foo19            down
# mmstartup -N foo19
Fri Nov 21 15:04:58 MST 2014: mmstartup: Starting GPFS ...
# mmgetstate -N foo19

 Node number  Node name        GPFS state 
------------------------------------------
      26      foo19            active

Multiple nodes at once (batch).

# cat newnodes.txt 
foo20
foo21
foo22
foo23
foo24
foo25
foo26
foo27
foo28
foo29
foo30
foo31
foo32
foo33
foo34
foo35
foo36
foo37
foo38
foo39
foo40
foo41
foo42
# mmaddnode -N newnodes.txt 
Wed foo  3 17:15:01 MST 2014: mmaddnode: Processing node foo20.example.org
Wed foo  3 17:15:04 MST 2014: mmaddnode: Processing node foo21.example.org
Wed foo  3 17:15:07 MST 2014: mmaddnode: Processing node foo22.example.org
Wed foo  3 17:15:09 MST 2014: mmaddnode: Processing node foo23.example.org
Wed foo  3 17:15:12 MST 2014: mmaddnode: Processing node foo24.example.org
Wed foo  3 17:15:15 MST 2014: mmaddnode: Processing node foo25.example.org
Wed foo  3 17:15:17 MST 2014: mmaddnode: Processing node foo26.example.org
Wed foo  3 17:15:20 MST 2014: mmaddnode: Processing node foo27.example.org
Wed foo  3 17:15:23 MST 2014: mmaddnode: Processing node foo28.example.org
Wed foo  3 17:15:25 MST 2014: mmaddnode: Processing node foo29.example.org
Wed foo  3 17:15:28 MST 2014: mmaddnode: Processing node foo30.example.org
Wed foo  3 17:15:31 MST 2014: mmaddnode: Processing node foo31.example.org
Wed foo  3 17:15:33 MST 2014: mmaddnode: Processing node foo32.example.org
Wed foo  3 17:15:36 MST 2014: mmaddnode: Processing node foo33.example.org
Wed foo  3 17:15:39 MST 2014: mmaddnode: Processing node foo34.example.org
Wed foo  3 17:15:41 MST 2014: mmaddnode: Processing node foo35.example.org
Wed foo  3 17:15:44 MST 2014: mmaddnode: Processing node foo36.example.org
Wed foo  3 17:15:47 MST 2014: mmaddnode: Processing node foo37.example.org
Wed foo  3 17:15:49 MST 2014: mmaddnode: Processing node foo38.example.org
Wed foo  3 17:15:53 MST 2014: mmaddnode: Processing node foo39.example.org
Wed foo  3 17:15:55 MST 2014: mmaddnode: Processing node foo40.example.org
Wed foo  3 17:15:58 MST 2014: mmaddnode: Processing node foo41.example.org
Wed foo  3 17:16:01 MST 2014: mmaddnode: Processing node foo42.example.org
mmaddnode: Command successfully completed
mmaddnode: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmaddnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
# mmchlicense client --accept -N newnodes.txt 

The following nodes will be designated as possessing GPFS client licenses:
    foo20.example.org
    foo21.example.org
    foo22.example.org
    foo23.example.org
    foo24.example.org
    foo25.example.org
    foo26.example.org
    foo27.example.org
    foo28.example.org
    foo29.example.org
    foo30.example.org
    foo31.example.org
    foo32.example.org
    foo33.example.org
    foo34.example.org
    foo35.example.org
    foo36.example.org
    foo37.example.org
    foo38.example.org
    foo39.example.org
    foo40.example.org
    foo41.example.org
    foo42.example.org
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@foonsd3 ~]# mmstartup -N newnodes.txt 
Wed foo  3 17:20:09 MST 2014: mmstartup: Starting GPFS ...
[root@foonsd3 ~]# mmgetstate -N newnodes.txt 

 Node number  Node name        GPFS state 
------------------------------------------
      27      foo20            active
      28      foo21            active
      29      foo22            active
      30      foo23            active
      31      foo24            active
      32      foo25            active
      33      foo26            active
      34      foo27            active
      35      foo28            active
      36      foo29            active
      37      foo30            active
      38      foo31            active
      39      foo32            active
      40      foo33            active
      41      foo34            active
      42      foo35            active
      43      foo36            active
      44      foo37            active
      45      foo38            active
      46      foo39            active
      47      foo40            active
      48      foo41            active
      49      foo42            active

Permanently removing a disk from a mdadm raid1 (mirror) array

| 1 Comment

A “special snowflake” system that I’m responsible for was configured with two SAS disks in mdadm raid1 arrays. Multiple times over the course of a few years the system would essentially dead lock (still responding to ICMP echo requests) and print errors on the console about I/O errors to [0:0:1:0]. There would be no other response on the console and this state would persist until forcibly power cycled via switched PDU. Upon a boot and a fsck cycle the system would be back in an apparently normal state.

dsas3-console-screenshot-2014-03-30
Continue Reading →