Additonal blockdevices or GPFS “disks” can be added to an existing filesystem. Each new block device first needs to as an NSD and then those NSDs can be added to a filessytem.

The state of the filesystem in terms of disks at start:

[root@foonsd1 init.d]# mmlsnsd

 File system   Disk name    NSD servers                                    
---------------------------------------------------------------------------
 foodata1      nsd1         foonsd1.tuc.noao.edu     
 foodata1      nsd2         foonsd2.tuc.noao.edu     
[root@foonsd1 init.d]# mmlsdisk foodata1
disk         driver   sector failure holds    holds
storage
name         type       size   group metadata data  status        availability
pool
------------ -------- ------ ------- -------- ----- ------------- ------------
------------
nsd1         nsd         512       1 Yes      Yes   ready         up
system       
nsd2         nsd         512       2 Yes      Yes   ready         up
system
[root@foonsd1 ~]# df -h /net/foodata1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/foodata1          18T  1.6T   16T   9% /net/foodata1       

There are a coule of different syntaxes that can be used to setup the NSDs. I opted for extended the original “DescFile” that was used to setup the filesystem. The /dev/sdc devices (nsd[34]) are new while the other two disks existed at filesystem creation time.

[root@foonsd1 init.d]# cat >descfile.txt< sdb:foonsd1::dataAndMetadata:1:nsd1:
> sdc:foonsd1::dataAndMetadata:1:nsd3:
> sdb:foonsd2::dataAndMetadata:2:nsd2:
> sdc:foonsd2::dataAndMetadata:2:nsd4:
> END
[root@foonsd1 init.d]# mmcrnsd -F ./descfile.txt
mmcrnsd: Processing disk sdb
mmcrnsd: Disk name nsd1 is already registered for use by GPFS.
mmcrnsd: Processing disk sdc
mmcrnsd: Processing disk sdb
mmcrnsd: Disk name nsd2 is already registered for use by GPFS.
mmcrnsd: Processing disk sdc
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@foonsd1 init.d]# mmlsnsd

 File system   Disk name    NSD servers                                    
---------------------------------------------------------------------------
 foodata1      nsd1         foonsd1.tuc.noao.edu     
 foodata1      nsd2         foonsd2.tuc.noao.edu     
 (free disk)   nsd3         foonsd1.tuc.noao.edu     
 (free disk)   nsd4         foonsd2.tuc.noao.edu     

Note that in the rewritten DescFile that the pre-existing nsds 1 & 2 are commented out.

[root@foonsd1 init.d]# cat descfile.txt
# sdb:foonsd1::dataAndMetadata:1:nsd1:
# sdc:foonsd1::dataAndMetadata:1:nsd3:
nsd3:::dataAndMetadata:1::system
# sdb:foonsd2::dataAndMetadata:2:nsd2:
# sdc:foonsd2::dataAndMetadata:2:nsd4:
nsd4:::dataAndMetadata:2::system

Now we just need to add the new NSDs to the filesystem.

[root@foonsd1 init.d]# mmadddisk foodata1 -F ./descfile.txt 

The following disks of foodata1 will be formatted on node
foonsd2.tuc.noao.edu:
    nsd3: size 9368502272 KB
    nsd4: size 9368502272 KB
Extending Allocation Map
Checking Allocation Map for storage pool 'system'
Completed adding disks to file system foodata1.
mmadddisk: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@foonsd1 init.d]# mmlsnsd 

 File system   Disk name    NSD servers                                    
---------------------------------------------------------------------------
 foodata1      nsd1         foonsd1.tuc.noao.edu     
 foodata1      nsd2         foonsd2.tuc.noao.edu     
 foodata1      nsd3         foonsd1.tuc.noao.edu     
 foodata1      nsd4         foonsd2.tuc.noao.edu     

[root@foonsd1 init.d]# mmlsdisk foodata1
disk         driver   sector failure holds    holds
storage
name         type       size   group metadata data  status        availability
pool
------------ -------- ------ ------- -------- ----- ------------- ------------------------
nsd1         nsd         512       1 Yes      Yes   ready         up
system       
nsd2         nsd         512       2 Yes      Yes   ready         up
system       
nsd3         nsd         512       1 Yes      Yes   ready         up
system       
nsd4         nsd         512       2 Yes      Yes   ready         up
system       
[root@foonsd2 ~]# df -h /net/foodata1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/foodata1          35T  1.5T   34T   5% /net/foodata1

At this point it’s not a bad idea to restripe the filesystem so existing data can benefit from the potental performance advantage of being distributed across additional disks.

[root@foonsd1 init.d]# mmrestripefs foodata1 -R
Scanning file system metadata, phase 1 ... 
Scan completed successfully.
Scanning file system metadata, phase 2 ... 
Scan completed successfully.
Scanning file system metadata, phase 3 ... 
Scan completed successfully.
Scanning file system metadata, phase 4 ... 
Scan completed successfully.
Scanning user file metadata ...
 100.00 % complete on Wed Jul 25 17:23:27 2012
Scan completed successfully.