Manual Resource group movement while clstrmgrES state is ST_INIT

                                    clstrmgrES  state  ST_INIT

                       

Today i am going to discuss clstrmgrES (cluster manager) state ST_INIT and how it affect while moving resource group from secondary node to primary node.
Let see different state of cluster manager and understand their purpose.

ST_INIT: cluster configured and down

ST_STABLE: cluster services are running with managed resources and cluster is in stable state.

while handling AIX hacmp cluster incident where one node from cluster rebooted which was primary node then resource group on primary node fail-over to secondary node.but after reboot of primary server we found that resource group was not fallback to primary node,so we started analysis on it and decided to to manual resource group switch.

while doing manual switch we followed following steps using smitty hacmp

Here for understanding purpose i am taking following node name and resource group name.

hacmp node name : aixhanode1 and aixhanode2 and 

resource group name : aixRG1

so in our scenario before reboot of aixhanode1 resource group  aixRG1 was on aixhanode1,but after reboot it failover to secondary node aixhanode2.so now client want to move aixRG1 resource group back on primary node aixhanode1.

so for manual resource group switch we followed below approach.

#clRGinfo

 -----------------------------------------------------------------------------

 Group Name  State    Node

 -----------------------------------------------------------------------------

 aixRG1      OFFLINE    aixhanode1

                    ONLINE     aixhanode2

1.smitty hacmp

 2. System Management (C-SPOC)

    3.HACMP Resource Group and Application Management

      4.Move a Resource Group to Another Node / Site

       5. Move Resource Groups to Another Node

          6.Choose RG     Here choose resource group name aixRG1

           7. Choose target node  but for step 7 we not able to see target node name in smitty panel  ???

so whats next ?? if you want to do cluster resource group switch then target node must be shown in smitty panel.after doing error analysis found that clstrmgrES  state must be ST_STABLE on both node,so after executing below command on both node we found that on primary node aixhanode1 it was in ST_INIT .

#aixhanode1#lssrc -ls clstrmgrES |grep state

Current state: ST_INIT    here is the issue this state must in ST_STABLE to move resource group from aixhanode2 to aixhanode1.

#aixhanode2#lssrc -ls clstrmgrES |grep state

Current state: ST_STABLE

so after identifying cause we simply started clstrmgrES on node aixhanode1 by using following method .

1.smitty hacmp

 2. System Management (C-SPOC)

  3.PowerHA SystemMirror Services

   4.Start Cluster Services

     5.

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

                                                     [Entry Fields]

* Start now, on system restart or both                now                                                                                                            +

  Start Cluster Services on these nodes              [aixhanode1]                                                                                                      +

* Manage Resource Groups                              Automatically                                                                                                  +

  BROADCAST message at startup?                       false                                                                                                          +

  Startup Cluster Information Daemon?                 true                                                                                                           +

  Ignore verification errors?                         false                                                                                                          +

  Automatically correct errors found during           Interactively                                                                                                  +

  cluster start?

Press enter wait for some time to command complete.

so after command completion we checked clstrmgrES and now its in stable state on aixhanode1,so now we can move resource group from aixhanode2 to aixhanode1.

#aixhanode1#lssrc -ls clstrmgrES |grep state

Current state: ST_STABLE

so we again follow below approach for moving aixRG1 to primary node .

1.smitty hacmp

 2. System Management (C-SPOC)

    3.HACMP Resource Group and Application Management

      4.Move a Resource Group to Another Node / Site

       5. Move Resource Groups to Another Node

          6.Choose RG     Here choose resource group name aixRG1

           7. Choose target node   and now we able to see traget node name "aixhanode1"

now press enter wait for some time.

after this check whether resource is online or not on primary node by clRGinfo command.

#clRGinfo

 -----------------------------------------------------------------------------

 Group Name  State    Node

 -----------------------------------------------------------------------------

 aixRG1       ONLINE   aixhanode1

              OFFLINE  aixhanode2

at last we can conclude that while moving resource group ,clstrmgrES state must be in ST_STABLE on both node to move resource group from one node to other.if its not in stable bring it in stable state and then only do resource group switch.

Thanks !!!