Solaris Cluster Administration

General Solaris

This document is provided to give a quick overview of the common administration commands used to administer Solaris and the Solaris Cluster software.

Here are some general Solaris commands:

Basic Commands

prtconf = displays out detailed hardware information such as memory, cards, and architecture.
prstat = displays active process statistics, with the top process taking the most resources.
prtdiag = displays very detailed hardware information, such as speed of CPUs, CPU cache, and which slots memory chips are installed. This command is platform specific.

User Management

To add a user <user> with access to /usr/local/<user>, make a home dir, and give that user a BASH shell:

useradd -d /usr/local/<user> -m -s /bin/bash <user>

Where: -d = the user's directory.
-m = make home directory.
-s = shell.
<user> = username.

Solaris administrators typically store user directories in /export/home/<user> because the /home directory is used by the Solaris Automounter. The Automounter gives the users the ability to login to many different machines and automatically have their home directory mounted on that machine in the /home area.

Networking

To set the name of the server:

vi /etc/nodename

To setup DNS on the server:

vi /etc/nsswitch.conf

Look for the line that starts with "hosts:" and add "dns" to the end of this line. You can also add "dns" to the beginning of that line, changing the order of name lookups

Add existing DNS servers to the Solaris server:

vi /etc/resolv.conf

To add a static host to the server:

vi /etc/hosts
vi /etc/net/ticlts/hosts
vi /etc/net/ticots/hosts
vi /etc/net/ticotsord/hosts

Interface Hostnames
On Solaris 10, all interfaces have a specific hostname that is usually created at the time of configuring IP. These files are under /etc/hostname.<NIC Name> and can be edited to make changes to the IPMP.

Changing Netmasks

vi /etc/inet/netmasks

Show Routes

netstat -rn

Create Bonds (Aggregates) and VLANS Use the dladm command to create aggregate links and VLAN information:

/usr/sbin/dladm create-aggr -P L2 -l active -d e1000g0 -d nge0 33

Where:
-P = Protocol (L2 or L3)
-l = LACP mode (Active or Passive)
-d = Interface Instance
33 = VLAN number
This stores persistent link aggregation information in /etc/aggregation.conf

Useful Directories

/usr/sadm = Solaris admin tools (smc).
/usr/sfw = Sun Free Ware - All the "free" Linux like tools.
/usr/ucb = BSD Tools.
/usr/X11 = X11 Gui Tools.

/var/adm = Log Files.
/var/log = Log Files Also.

/opt = Commercial and Distributed Software.
/net = Automount point - Do not use unless the automounter is disabled.
/home = Automount point - Do not use unless the automounter is disabled.
/proc = Processes.

Package Management

Solaris uses software called "packages" which are standard System-V PKG files. Packages come in two varieties: Filesystem or Datastream. The Filesystem package is what you will find on the Solaris Software CDs, like the Cluster DVD. Datastreams is a filesystem package that is rolled into a single file, making it easy to compress and distribute. Packages use a common naming conventions of ORG-software, such as SUNWspro, where SUNW is Sun's Stock Ticker Symbol and spro is the software package.

To add a package in Solaris:

gunzip -d <package name> (optional)
pkgadd -d <package name>

To use the Blastwave software updater (equivalent to Debian's apt-get):

pkg-get -i <package name>

Managing Services

Solaris 10 did away with the standard "RC" scripts that you might be used to having on Linux systems. Instead, they went with Solaris Management Facility (SMF), which uses svcs to view services, svcadm to administer services, and svccfg to configure services.

To enable a service

svcadm enable samba

To disable (stop) a service

svcadm disable samba

Managing Disks

This information only applies to UFS file systems, for the most part. Solaris uses format to administer disks. A Slice is what Solaris calls a partition. The disk naming convention is c0t0d0s0 where that translates to "Controller 0, Target 0, LUN 0, Slice 0". It is Solaris tradition to use slice 2 as a representation of the full disk and is not used for other purposes. USB and removable devices are managed through mformat instead of format. CDROMs are automatically mounted under /cdrom.

To view error counters:

iostat -En

To view throughput

iostat -xn 1

iSCSI

To enable iSCSI

svcadm enable iscsitgt

To set iSCSI discovery to Static

iscsiadm modify discovery -s enable

To add a static mapping in iSCSI

iscsiadm add static-config <iqn.number>,<host>

Solaris Cluster

These commands are specific to the SUN Cluster Software, which is the mechanism for providing High Availability to specific services within the cluster. The terminology used are the following convections:

Cluster = The series of servers that collectively provide services with a failover.
Node = A member of the cluster.
Resource Group = A collection of HA resources which are managed as a group.
Network Resource = In Solaris world, a Network Resource is a Virtual IP address, or at least the mechanism to create a VIP.
Data Resource = A service that is intended to be managed by the cluster.

Cluster Command Summary

These are some common commands that are used to administer the Cluster:

Entire Cluster Specific:
cluster list-cmds = List of utilities to manage the Cluster.
cluster show = Very detailed output of the Cluster's status.
cluster status = List the health status of the Cluster.

Data Resource Specific:
clresource clear = Clear error counters.
clresource DISABLE / ENABLE = Disable or Enable resource.
clresource MONITOR / UNMONITOR = Enable or disable the Cluster's ability to tell whether or not that resource is down.
clresource list = List the status of each resource in the Cluster.
clresource show = Very detailed output of the Cluster's resources.
clresource List the Online / Offline status of each resource.

Device Specific:
cldevice check = Check to ensure device consistency.
cldevice clear = Clear all DID references to stale devices.
cldevice list = List cluster devices.
cldevice MONITOR / UNMONITOR = Enable / Disable monitoring of disk paths.
cldevice populate = Populate the global-devices namespace.
cldevice show = Show device properties.
cldevice status = Display the status of monitored devices.

Node Specific:
clnode list = List the cluster nodes.
clnode show = Display cluster nodes and their properties.
clnode status = Display the status of cluster nodes.

Quorum Specific:
clquorum list = List quorum devices.
clquorum ENABLE / DISABLE = Put quorum device into maintenance mode.
clquorum reset = Reset the quorum configuration.
clquorum show = Show quorum devices and their properties.
clquorum status = Display the status of the cluster quorum.

Network Resource Specific:
clreslogicalhostname list = List resources.
clreslogicalhostname MONITOR / UNMONITOR = Enable / Disable monitoring of resources.
clreslogicalhostname show = Show resources and their properties.
clreslogicalhostname status = Display the status of resources.

Menu Driven Configuration:
clsetup = Cluster Setup Menu.

Cluster Configuration Specific:
scconf -p = Verify Cluster configuration information.
scconf -c -q <node>=,maintstate = Place a node in maintenance mode.
scconf -c -q reset = Take a node out of maintenance mode.

Cluster DID Administration Specific:
scdidadm -l = List device mappings for this host.
scdidadm -L = List device mappings for all hosts.
scdidadm -r = Reconfigure DID, search for all disks.
scdidadm -c = Check the configuration file against disks.

scshutdown -y -g 15 = Shutdown the whole Cluster.

scstat = Display detailed information about the Cluster.
scstat -q = Display quorum vote information.
scstat -D = Display existing device group resources in the Cluster.
scstat -p = Display the Cluster components.
scstat -i = Display the status of the Cluster's public networks.

Switching Resources In A Cluster
scswitch -F -g <resource group> = Shut down a resource group.
scswitch -Z -g <resource group> = Bring up a resource group.
scswitch -z -g <resource group> -h <node name> = Move failover resource group to another node.
scswitch -R -h <node name> -g <resource group> = Restart resource group.
scswitch -S -h <node name> = Evacuate all resources from a node.
scswitch -n -j <data or network resource> = Disable resource from resource group.
scswitch -e -j <data or network resource> = Enable resource from resource group.
scswitch -c -j <data or network resource> -h <node name> -f STOP_FAILED = Clear STOP_FAILED on a resource.
scswitch -m -D = Place Cluster in maintenance mode.
scswitch -z -D -h = Take a Cluster out of maintenance mode.

Cluster Mode

The Cluster can be in one of a few "modes" as well as components that make up the resources within the cluster can also operate in "modes"

Boot into Non-Cluster Mode

This specific mode is made by editing the GRUB script in one of the following ways:

Preferred
Prior to rebooting the Node, edit the /boot/grub/menu.lst; rem out the "top" portion and un-rem the bottom:

vi /boot/grub/menu.lst
----Have it look like this for Non-Cluster Mode----
#title Solaris 10 5/09 s10x_u7wos_08 X86
#findroot (rootfs0,0,a)
#kernel /platform/i86pc/multiboot -v
#module /platform/i86pc/boot_archive
# Enable below and disable above to boot in non-cluster mode
title Solaris 10 5/09 s10x_u7wos_08 X86
findroot (rootfs0,0,a)
kernel /platform/i86pc/multiboot -x
module /platform/i86pc/boot_archive

After this change is made, reboot the server and it will boot into "non-cluster" mode.

shutdown -y -g0 -i6

Or you can reverse that file to boot back into Cluster Mode:

title Solaris 10 5/09 s10x_u7wos_08 X86
findroot (rootfs0,0,a)
kernel /platform/i86pc/multiboot -v
module /platform/i86pc/boot_archive
# Enable below and disable above to boot in non-cluster mode
#title Solaris 10 5/09 s10x_u7wos_08 X86
#findroot (rootfs0,0,a)
#kernel /platform/i86pc/multiboot -x
#module /platform/i86pc/boot_archive

From the Console

Hit ENTER at the GRUB Console, "e" to edit the config file, edit the 2nd line from the Top, "e", and have it look like this:

kernel /platform/i86pc/multiboot -v -x

Install Mode

If you have installed the software, you have to issue the following command to get the cluster out of "install mode":

scconf -c -q installmodeoff

Online / Offline

The cluster can also operate in an ON and OFF line mode in general, as well as resources and resource groups can have these properties. Typically, the cluster will operate in ONLINE (both nodes) Mode, with all physical shared disk resources ONLINE on both nodes, but only one node will have the resources from a resource group ONLINE at any given time.

To OFFLINE a resource group:

clresource disable <resource group>

To ONLINE a resource group:

clresource enable <resource group>

Common Cluster Operations

Here are some common operations that will be employed to manage the cluster:

Shutdown a Cluster

1. On any node:

/usr/cluster/bin/cluster shutdown -g0 -y

2. Verify that the cluster is down:

/usr/cluster/bin/cluster status -t <node>

Shutdown a Clustered Node

1. On the node you want shutdown:

/usr/cluster/bin/clnode evacuate <node>

2. Shutdown that node:

shutdown -g0 -y -i0

Change The Active Node For A Resource Group

Or to failover to the other node:

/usr/cluster/bin/clresourcegroup switch -n <node name> <resource group>

To Add NFS Resource and HA Storage Resource For A Server

Here are some detailed instructions on how to add a HA Storage Volume to a server.

Disk Operations

1. Create a volume using the SAN's "front end" interface:

a. Use the naming convention <server pretext name>-<function>-<disk#>.
b. Restrict access by IP, use 192.168.33.11 (NFS11) and 192.168.33.72 (NFS12).
c. No Thin Provisioning.
d. Make the volume larger then the intended size for use, with 0% reserve for snapshots.
e. Enabled shared access from multiple initiators.

2. Create the iSCSI Initiators on the nodes:

iscsiadm add static-config <iqn.number>,<iSCSI Target Server IP>

3. Ensure that the iSCSI targets are listed on the host:

iscsiadm list target
----Output----
Target: iqn.2001-05.com.equallogic:0-8a0906-2b92b3b04-f66000000404ae5f-zfs-idx-mls-d0
        Alias: zfs-idx-mls-d0
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1

4. Rescan the DID list:

scdidadm -r

5. Check to see if the iSCSI Initiator made it to the Cluster's DID list:

/usr/cluster/bin/cldevice list -v

6. Make a Zpool from the iSCSI connections, format it as ZFS and create a mount point:

zpool create -f <pool name> <disk>
zfs create -o mountpoint=<mount point> <zpool>/<zfs file system>

Resource Group Operations

Now that we have added a disk to the hosts via iSCSI, we now need to configure the cluster service to utilize failover on this service.
1. Create a data resource for the Volume created in the previous instructions:

/usr/cluster/bin/clresource create -g nfs_group -t SUNW.HAStoragePlus -p Zpools=<your Zpool> HASP-<name of resource>

2. Create a file under /global/base/SUNW.nfs/ called dfstab.<NFS Resource Name>:

vi /global/base/SUNW.nfs/dfstab.nfs-myresource
----Input----
share -F nfs -o rw,anon=0 -d "<your Description> /<mountpoint>

3. Create a data resource for the NFS service we want to append to the volume we created:

/usr/cluster/bin/clresource -g nfs_group -t SUNW.nfs -p Resource_dependencies=HASP-<name of resource (from above, #1)> <nfs-resource-name>

4. Verify that the NFS service is sharing your intended volume:

share
----output----
-               /idx3-acnt   rw,anon=0   "HA IDX3-Acnt File Share" 
-               /idx-dev   rw,anon=0   "HA idx-stg Share" 
-               /zfstest   rw,anon=0   "HA test File Share" 
-               /idx-mls   rw,anon=0   "HA idx-mls File Share" 
-               /idx-stg   rw,anon=0   "HA idx-stg File Share" 
-               /data   rw,anon=0   "HA Sugar Data File Share" 
-               /nfsmail   rw,anon=0   "HA Mail File Share" 
-               /idx3-bin   rw,anon=0   "HA IDX3-Bin File Share" 
-               /prd-crm   rw,anon=0   "HA prd-crm File Share"

To Unshare a Mounted Filesystem HA Resource

Resource Group Operations

1. Determine what resource group has the data or network resource you want to disable:

/usr/cluster/bin/clresource list -v
----Output----
Resource Name       Resource Type            Resource Group
-------------       -------------            --------------
HASP-zfstest        SUNW.HAStoragePlus:6     nfs_group
HASP-nfsmail        SUNW.HAStoragePlus:6     nfs_group
nfs-fe-44           SUNW.LogicalHostname:3   nfs_group
nfs-fe-55           SUNW.LogicalHostname:3   nfs_group
nfs-mail            SUNW.nfs:3.2             nfs_group
nfs-test            SUNW.nfs:3.2             nfs_group
HASP-data           SUNW.HAStoragePlus:6     nfs_group
HASP-idx3-bin       SUNW.HAStoragePlus:6     nfs_group
nfs-idx3-bin        SUNW.nfs:3.2             nfs_group
HASP-idx3-acnt      SUNW.HAStoragePlus:6     nfs_group
nfs-idx3-acnt       SUNW.nfs:3.2             nfs_group
nfs-sugardata       SUNW.nfs:3.2             nfs_group
nfs-fe-99           SUNW.LogicalHostname:3   nfs_group
[root@nfs12] 08:28:47 /usr/cluster/bin

2. Disable the data or network resource:
In this example, there was only one resource group, NFS_GROUP. A resource name in the example below is "nfs-test".

/usr/cluster/bin/clresource disable <data or network resource name>

3. Determine whether the resource was disabled:

/usr/cluster/bin/clresource status -v
----Output----
nfs-test                  nfs11               Offline           Offline
                          nfs12               Offline           Offline

4. If this is an NFS resource, remove it from the /global/base/SUNW.fs/dfstab.<NFS-reource.name>
5. Delete the data or network resource:

/usr/cluster/bin/clresource delete <data or network resource name>

6. If this is an NFS resource, disable and delete the child HA Volume HA resource associated with this NFS resource:

/usr/cluster/bin/clresource disable <data resource>
/usr/cluster/bin/clresource delete <data resource>

7. Remove the cluster DID that references the HA Volume you are removing in step 6.

/usr/cluster/bin/cldevice show -v
----Output----
DID Device Name:                                /dev/did/rdsk/d6
  Full Device Path:                                nfs11:/dev/rdsk/c2t6090A048B0B3624C60ADC40100006030d0
  Full Device Path:                                nfs12:/dev/rdsk/c2t6090A048B0B3624C60ADC40100006030d0
  Replication:                                     none
  default_fencing:                                 global
  Disk ID:                                         6090a048b0b3624c60adc40100006030
  Ascii Disk ID:                                   `

Disk Operations

1. List and Remove the static iSCSI mappings from each node:

iscsiadm list target
----Output----
Target: iqn.2001-05.com.equallogic:0-8a0906-1352b3b04-3d00000002c4ae1d-zfs-idx3-bin-d1
        Alias: zfs-idx3-bin-d1
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
Target: iqn.2001-05.com.equallogic:0-8a0906-0ce2b3b04-cd20000002a4ae1d-zfs-idx3-bin-d0
        Alias: zfs-idx3-bin-d0
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
---------------
iscsiadm remove static-config <iqn.number>

2. Update the DID data table:

/usr/cluster/bin/scdidadm -C

3. Confirm that the changes were made successfully:

/usr/cluster/bin/cldev list -v

To Add Space To An Existing Zpool Within The Cluster

1. Create a volume using the SAN's "front end" interface:

a. Use the naming convention <server pretext name>-<function>-<disk#>.
b. Restrict access by IP, use 192.168.33.11 (NFS11) and 192.168.33.72 (NFS12).
c. No Thin Provisioning.
d. Make the volume larger then the intended size for use, with 0% reserve for snapshots.
e. Enabled shared access from multiple initiators.

2. Add static iSCSI mapping to the created volume:

iscsiadm add static-config <iqn.number>,<iscsi target server's IP>

3. Ensure that your Zpool is in good health:

zpool status <zpool>

4. Make the cluster aware of the new volume:

scsiadm -r

5. Add the newley made volume to the pool:

zpool add <zpool> <disk>

ZFS Specific

Zetta File System is the new FS that Sun has developed. It automatically moved data from identified "bad" areas on the HD to known "good" areas of the storage pool. There are only two commands to manage ZFS:

ZPOOL

Zpool command is used to create and manage a "pool" or collection of storage resources, as well as a base ZFS filesystem, which is also named after the pool. All subsequent filesystems are created usign a different command but have to reside under the pool. Multiple pools can be created and are recommended to manage like-minded filesystems. Here are some common ZPOOL commands:

Dry Run

At any of these commands, you can use the -n flag (both zpool and zfs):

zpool create -n <pool> <disk1> <disk2>

Create a Pool

Raid0

zpool create <pool> <disk1> <disk2> ... <diskX>

Raid1

zpool create <pool> mirror <disk1> <disk2>

Raid5

zpool create <pool> raidz <disk1> <disk2> ... <diskX>

Double Parity

zpool create <pool> raidz2 <disk1> <disk2> <disk3>

Raid1+0

zpool create <pool> mirror <disk1> <disk2> mirror <disk3> <disk4>

Raid5+0

zpool create <pool> raidz <disk1> <disk2> <disk3> raidz <disk4> <disk5> <disk6>

Hybrid

zpool create <pool> mirror <disk1> <disk2> mirror <disk3> <disk4> raidz <disk5> <disk6> <disk7> <disk8> log <disk9> cache <disk10> spare <disk11> <disk12>

Destroy Pool

Normal:

zpool destroy <pool>

With Faulted Device:

zpool destroy -f <pool>

To recover a destroyed zpool:

zpool import -D

List Pool

To list the current Zpools, use the following:

zpool list

To view just certain items within the Zpool list context, use:

zpool list -o name,size

To view the history of commands used to administer the Zpool:

zpool histroy -l

To view the IOStats on the Zpool:

zpool iostat -v

To clear the counters on the Zpool:

zpool clear <pool>

Pool Health

Use the status zpool command to see the "health" of the zpool(s)

zpool status -v

To view a brief discription:

zpool status -x

Add and Remove Disks from Pool

You can add volumes to existing pools, making the space expand by the limit of storage you have added:

zpool add <pool> <redundancy option> <disk1> <disk2> ... <disk99>

Or you can remove disks from the pool using this format, but this only applies to SPARE and CACHE deivces:

zpool remove <pool> <disk1> <disk2> ... <disk99>

You can use the attach command to add a new device to an existing mirrored or non-mirrored device:

zpool attach <pool> <disk5> <disk6>

You can detach a device from a mirrored storage pool:

zpool detach <pool> <disk5>

Online / Offline

You can use the offline command to temporaily disconnect storage devices from a pool:

zpool offline <pool> <disk5>

You cannot take a pool offline to the point where it becomes faulted.
An offlined device is not detached from the storage pool.
By default, the pool remains offline after rebooting the system until explicitly told to "online" the device. Use the following to have the device come back online when the system is rebooted:

zpool offline -t <pool> <disk5>

To bring a device back on-line:

zpool online <pool> <disk5>

Pool Properties

Zpool, like ZFS, have global properties that can be set onto the pool. To list all of the properties of a zpool:

zpool get all <pool>

To just list a certain property:

zpool get <property> <pool>

To set properties:

zpool set <preperty>=<operation> <pool>

Here is a list of possible properties

Scrub Pool

This is a way to force a data integrty check on your ZFS file system. Scrub transverses all the data in the pool once and verifies that all blocks can be read. Scrub might impact the performance of running filesystems, so use sparingly:

zpool scrub <pool>

ZFS

The second command, ZFS, is used for managing individual filesystems within a Zpool.

Create / Destroy

To create a ZFS filesystem:

zfs create <pool>/<filesystem>

To create a ZFS filesystem with optional properties:

zfs create -o <property>=<value> <pool>/<filesystem>

To destroy a ZFS filesystem:

zfs destroy <pool>/<filesystem>

To rename a ZFS Filesystem:

zfs rename <pool>/<filesystem>

ZFS Properties

You can change and apply properties to the ZFS filesystems using the ZFS command without having to manage different software daemons. To set the properties on an existing filesystem, use:

zfs set <property>=<value> <pool>/<filesystem>

To see all the properties set onto a ZFS filesystem:

zfs get all <pool>

Here is a list of properties:

Properties that we will commonly use are:
mountpoint=<mount point>
compression=<on/off>
sharenfs=<on/off>

Mount Points

By default, all ZFS file systems are mounted by ZFS at boot by using:

svc://system/filesystem/local

File systems are mounted under /path, where path is the name of the file system.

To see all of your ZFS mountpoints:

zfs mount

To mount all of your ZFS filesystems:

zfs mount -a

To unmount ZFS filesytems:

zfs umount <pool>/<filesystem>

Snapshots

ZFS does snapshotting incrementally, which is created in <pool>/<filesystem>/.zfs/snapshot/<snap name>. The usage guidelines below:

Create a snapshot

zfs snapshot <pool>/<filesystem>@<snap name>

Create a snapshot that includes all decendant filesystems within a Zpool

zfs snapshot -r <pool>@<snap name>

Destroy a snapshot

zfs destroy <pool>/<filesystem>@<snap name>

Rename a snapshot

zfs rename <pool>/<filesystem>@<old snap name> <pool>/<filesystem>@<new snap name>

View snapshots

zfs list

zfs list -t snapshot

View snapshot space

zfs list -o space

List content of snapshot:

ls -l /<zfs File System name>/.zfs/snapshot/<snapshot name>

Copy back content of a snapshot:

cp /<zfs File System name>/.zfs/snapshot/<snapshot name>/<original content> /target/directory

Rollback Snapshot To roll back to the last snapshot:

zfs rollback

To roll back from a specific snapshot:

zfs rollback <pool>/<filesystem>@<snap name>

To roll back from a specific snapshot destroying all newer snapshots then the one restoring from:

zfs rollback -r <pool>/<filesystem>@<snap name>

NOTE: The ZFS volume will have to unmount itself to be able to rollback a snapshot.

Troubleshooting

Here are some situations that I got in the lab and how I resolved the issues.

Node Does Not Reboot After First Installing Sun Cluster

The console says that the boot archive is corrupt:

Login with root Password at the prompt
svcadm clear system/boot-archive

Host Key Verification Failure

If the cluster software stops installing, complaining about host key verification issues while trying to get the other NFS server to talk to the configured host (as below).

Host key verification failed.
ssh to "nfs02.lab.cisdata.net" failed

The hostnames are not quite similar: One solaris box is sunnfs01.lab.cisdata.net and the other is nfs02.lab.cisdata.net. Not a big deal unless you don't have DNS resolution (like in my lab) and you have to edit the /etc/hosts file incorrectly. At any rate, to resolve the problem, I did the following:

1. SSH to each box using the same hostname that you will use in the cluster config:

  Sunnfs01# ssh nfs02.lab.cisdata.net
cat ~/.ssh/knownhosts    <-- See if this looks correct.
exit

nfs02# ssh sunnfs01.lab.cisdata.net
cat ~/.ssh/knownhosts    <-- See if this looks correct.
exit

2. Put the other host's hostname into ~/.rhosts

vi ~/.rhosts
sunnfs01.lab.cisdata.net

---or---

nfs02.lab.cisdata.net

Cluster Amnesia

Amnesia occurs if all the nodes leave the cluster in staggered groups. An example is a two-node cluster with nodes A and B. If node A goes down, the configuration data in the CCR is updated on node B only, and not node A. If node B goes down at a later time, and if node A is rebooted, node A will be running with old contents of the CCR. This state is called amnesia and might lead to running a cluster with stale configuration information.

To recover from Amnesia:

Scenario: Two node cluster (NFS11 and NFS12) with one Quorum Device, NFS11 has gone bad, and amnesia protection is preventing NFS12 from booting up.

1. Boot NFS12 in non-cluster mode (-x).
2. Edit NFS12's /etc/cluster/ccr/golbal/infrastructure file:

vi /etc/cluster/ccr/global/infrastructure
cluster.properties.installmode = enabled
cluster.nodes.NFS11.properties.quorum_vote = 0
---Remove all entries with---
"cluster.quorum__devices"

3. Regen the CRC on the cluster config file (NFS12):

/usr/cluster/lib/sc/ccradm -i /etc/cluster/ccr/infrastructure -o

4. Reboot NFS12 in cluster mode.

Split Brain

Split brain occurs when the cluster interconnect between nodes is lost and the cluster becomes partitioned into subclusters, and each subcluster believes that it is the only partition.

How to Replace a Quorum Disk

To perform this operation, you need to follow the directions on how to create a quorum disk above. You (typically) cannot remove the last quorum device.

1. Configure a new quorum device on the SAN that currently includes the disk to be replaced.
2. Remove and replace the failed disk.
3. Verify the operation is successful:

scstat -q

How to Put The Quorum Disk into Maintenance Mode

Put a quorum device into maintenance state when taking the quorum device out of service for an extended period of time. This way, the disk device's quorum vote count is set to zero and does not contribute to the quorum count while the device is being serviced. While in maintenance state, the quorum device's configuration information is preserved.

1. Issue the following command:

scconf -c -q globaldev=d<x>,maintstate

2. Verify that the quorum device is now in maintenance mode:

scconf -p | grep -i quorum

To bring a quorum device out of maintenance mode:

1. Issue the following command:

scconf -c -q globaldev=d<x>,reset

2. If you are resetting the quorum count because a node was in maintenance state, reboot the node.
3. Verify the quorum vote count:

scconf -p | grep -i quorum

How To Remove the Last Quorum Device From A Cluster

If this is the last quorum device on a two-node cluster, the cluster must be placed into install mode before scconf allows you to remove the device from the configuration. This should only be done if a node is being removed from the cluster.

1. Place the cluster in install mode:

scconf -c -q installmode

2. Remove the quorum device using the scconf command:

scconf -r -q globaldev=d<x>

3. Verify that the quorum device has been removed:

scstat -q

How To Remove a Node From A Cluster

1. Reboot the node that you want to remove in non-cluster mode (-x)
2. On the node that you want to remove:

clnode remove -F

3. On the node that is still in the cluster:

clnode clear -F <nodename>

4. From another cluster node, verify the node removal:

clnode status <nodename>

How To Uninstall Sun Cluster Software From A Cluster Node

Perform this procedure to uninstall Sun Cluster software from a global-cluster node before you disconnect it from a fully established cluster configuration. You can use this procedure to uninstall software from the last remaining node of a cluster.

1. Ensure that you have removed the node from the cluster configuration by using clnode remove before you continue with this procedure.
2. From the active cluster member, add the node that you intend to uninstall to the cluster's node authentication list.

claccess allow -h <hostname>

3. Reboot the global-cluster node into noncluster mode.

shutdown -g0 -y -i0
b -x

4. In the /etc/vfstab file, remove all globally mounted file-system entries except the /global/.devices global mounts.
5. If you intend to reinstall Sun Cluster software on this node, remove the Sun Cluster entry from the Sun Java Enterprise System (Java ES) product registry.

/var/sadm/prod/SUNWentsysver/uninstall

6. If you do not intend to reinstall the Sun Cluster software on this cluster, disconnect the transport cables and the transport switch, if any, from the other cluster devices.

Troubleshooting a Node Uninstallation

Unremoved Cluster File System Entries

The following error messages indicate that the global-cluster node you removed still has cluster file systems referenced in its vfstab file.

Verifying that no unexpected global mounts remain in /etc/vfstab ... failed
scinstall:  global-mount1 is still configured as a global mount.
scinstall:  global-mount1 is still configured as a global mount.
scinstall:  /global/dg1 is still configured as a global mount.
 
scinstall:  It is not safe to uninstall with these outstanding errors.
scinstall:  Refer to the documentation for complete uninstall instructions.
scinstall:  Uninstall failed.

To correct this error, return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure. Rerun the "clnode remove" command.

Unremoved Listing in Device Groups

The following error messages indicate that the node you removed is still listed with a device group.

Verifying that no device services still reference this node ... failed
scinstall:  This node is still configured to host device service "
service".
scinstall:  This node is still configured to host device service "
service2".
scinstall:  This node is still configured to host device service "
service3".
scinstall:  This node is still configured to host device service "
dg1".
 
scinstall:  It is not safe to uninstall with these outstanding errors.          
scinstall:  Refer to the documentation for complete uninstall instructions.
scinstall:  Uninstall failed.

How To Enable an SNMP Event MIB

This procedure shows how to enable an SNMP event MIB:

clsnmpmib enable [-n node] MIB

---To send to a trap receiver---

clsnmphost add -c SNMPcommunity [-n node] host

---To add a SNMP user to the node---

clsnmpuser create -n node -a authentication \ -f password user

--Scotts 14:49, 29 October 2009 (UTC)

Solaris Cluster Administration

Contents

General Solaris

Basic Commands

User Management

Networking

Useful Directories

Package Management

Managing Services

Managing Disks

iSCSI

Solaris Cluster

Cluster Command Summary

Cluster Mode

Boot into Non-Cluster Mode

Install Mode

Online / Offline

Common Cluster Operations

Shutdown a Cluster

Shutdown a Clustered Node

Change The Active Node For A Resource Group

To Add NFS Resource and HA Storage Resource For A Server

Disk Operations

Resource Group Operations

To Unshare a Mounted Filesystem HA Resource

Resource Group Operations

Disk Operations

To Add Space To An Existing Zpool Within The Cluster

ZFS Specific

ZPOOL

Dry Run

Create a Pool

Destroy Pool

List Pool

Pool Health

Add and Remove Disks from Pool

Online / Offline

Pool Properties

Scrub Pool

ZFS

Create / Destroy

ZFS Properties

Mount Points

Snapshots

Troubleshooting

Node Does Not Reboot After First Installing Sun Cluster

Host Key Verification Failure

Cluster Amnesia

Split Brain

How to Replace a Quorum Disk

How to Put The Quorum Disk into Maintenance Mode

How To Remove the Last Quorum Device From A Cluster

How To Remove a Node From A Cluster

How To Uninstall Sun Cluster Software From A Cluster Node

Troubleshooting a Node Uninstallation

How To Enable an SNMP Event MIB