Oracle Apps DBA

My photo
More than 5+ Years of IT Experience in administering Enterprise wide Multi Vendor UNIX Servers, Oracle Databases (8i to 11gR2), Middle-tiers, Applications and Clusters. I am a Sun Certified System Administrator (SCSA) for Solaris 10, oracle 10g database certified associate(OCA), Oracle 10g Database certified professional (OCP) and Oracle E-Business suite R12 certified professionl(OCP)

Oracle Database and Applications

Saturday, 19 November 2011

Manage startup/shutdown/status services in 11gR2 RAC

Hi,

Earlier i have written one article to manage Grid infra home on standalone Installation. Now i am writing the below article - How to Manage startup/shutdown/status of Grid Infrastructure software in 11gR2 RAC Environment.

All these commands are demonstrated on a 4 node cluster running on OEL 5.6.

How to check the status of CRS on specific node:



[ora11g@racnode1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online




Note: crsctl command will be present in GRID_INRA_HOME


How to check the status of services on all nodes:




[ora11g@racnode1 ~]$ srvctl status nodeapps
VIP racnode1-vip is enabled
VIP racnode1-vip is running on node: racnode1
VIP racnode2-vip is enabled
VIP racnode2-vip is running on node: racnode2
VIP racnode3-vip is enabled
VIP racnode3-vip is running on node: racnode3
VIP racnode4-vip is enabled
VIP racnode4-vip is running on node: racnode4
Network is enabled
Network is running on node: racnode1
Network is running on node: racnode2
Network is running on node: racnode3
Network is running on node: racnode4
GSD is disabled
GSD is not running on node: racnode1
GSD is not running on node: racnode2
GSD is not running on node: racnode3
GSD is not running on node: racnode4
ONS is enabled
ONS daemon is running on node: racnode1
ONS daemon is running on node: racnode2
ONS daemon is running on node: racnode3
ONS daemon is running on node: racnode4
eONS is enabled
eONS daemon is running on node: racnode1
eONS daemon is running on node: racnode2
eONS daemon is running on node: racnode3
eONS daemon is running on node: racnode4
[ora11g@racnode1 ~]$



How to check the status of complete clusterware stack on all nodes:



[ora11g@racnode1 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMDATA1.dg
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
ora.LISTENER.lsnr
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
ora.asm
ONLINE ONLINE racnode1 Started
ONLINE ONLINE racnode2 Started
ONLINE ONLINE racnode3 Started
ONLINE ONLINE racnode4 Started
ora.eons
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
ora.gsd
OFFLINE OFFLINE racnode1
OFFLINE OFFLINE racnode2
OFFLINE OFFLINE racnode3
OFFLINE OFFLINE racnode4
ora.net1.network
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
ora.ons
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
ora.registry.acfs
ONLINE ONLINE racnode1
ONLINE ONLINE racnode2
ONLINE ONLINE racnode3
ONLINE ONLINE racnode4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE racnode2
ora.oc4j
1 OFFLINE OFFLINE
ora.racdb.db
1 OFFLINE OFFLINE
2 OFFLINE OFFLINE
3 OFFLINE OFFLINE
4 OFFLINE OFFLINE
ora.racnode1.vip
1 ONLINE ONLINE racnode1
ora.racnode2.vip
1 ONLINE ONLINE racnode2
ora.racnode3.vip
1 ONLINE ONLINE racnode3
ora.racnode4.vip
1 ONLINE ONLINE racnode4
ora.scan1.vip
1 ONLINE ONLINE racnode2
[ora11g@racnode1 ~]$




How to check the status of clusterware servers:




[ora11g@racnode1 ~]$ crsctl status server -f
NAME=racnode1
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode2
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode3
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode4
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=



How to stop csr on all nodes of clusterware:



[root@racnode1 bin]# ./crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode3'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode4'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode4'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode4'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode4'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode3' succeeded
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'racnode2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode2'
CRS-2673: Attempting to stop 'ora.racnode3.vip' on 'racnode3'
CRS-2677: Stop of 'ora.racnode3.vip' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode1.vip' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.racnode1.vip' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode4.vip' on 'racnode4'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.racnode2.vip' on 'racnode2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'racnode2'
CRS-2677: Stop of 'ora.racnode4.vip' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.racnode2.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode3'
CRS-2677: Stop of 'ora.asm' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode3'
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode4'
CRS-2677: Stop of 'ora.asm' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'racnode4'
CRS-2673: Attempting to stop 'ora.eons' on 'racnode4'
CRS-2677: Stop of 'ora.ons' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode3'
CRS-2677: Stop of 'ora.net1.network' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.ons' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode4'
CRS-2677: Stop of 'ora.net1.network' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode2'
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode1'
CRS-2677: Stop of 'ora.ons' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode1'
CRS-2677: Stop of 'ora.net1.network' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode3' has completed
CRS-2677: Stop of 'ora.eons' on 'racnode4' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode4' has completed
CRS-2677: Stop of 'ora.crsd' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode3'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.crsd' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode4'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode4'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode1' has completed
CRS-2673: Attempting to stop 'ora.eons' on 'racnode2'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode2'
CRS-2677: Stop of 'ora.evmd' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ons' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode2'
CRS-2677: Stop of 'ora.net1.network' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode2' has completed
CRS-2677: Stop of 'ora.asm' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode3'
CRS-2677: Stop of 'ora.asm' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode4'
CRS-2677: Stop of 'ora.crsd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode3'
CRS-2677: Stop of 'ora.cssd' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode4'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.evmd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode1'
CRS-2677: Stop of 'ora.diskmon' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode2'
CRS-2677: Stop of 'ora.cssd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode2'
CRS-2677: Stop of 'ora.diskmon' on 'racnode2' succeeded
[root@racnode1 bin]#



How to stop crs on specific node:



[root@racnode1 bin]# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode1.vip' on 'racnode1'
CRS-2677: Stop of 'ora.racnode1.vip' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.racnode1.vip' on 'racnode2'
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode1' succeeded
CRS-2676: Start of 'ora.racnode1.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode1'
CRS-2677: Stop of 'ora.ons' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode1'
CRS-2677: Stop of 'ora.net1.network' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode1' has completed
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@racnode1 bin]#




How to start crs on specific node:




[root@racnode1 bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.




How to check the status of specific Instance across cluster nodes:



[ora11g@racnode1 ~]$ srvctl status instance -d racdb -i racdb1
Instance racdb1 is running on node racnode1




How to check the status of all instances across the cluster nodes:



[ora11g@racnode1 ~]$ srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4



How to stop specific Instance across cluster nodes:



[ora11g@racnode1 ~]$ srvctl stop instance -d racdb -i racdb4
[ora11g@racnode1 ~]$ srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is not running on node racnode4
[ora11g@racnode1 ~]$




How to start specific Instance across cluster nodes:




[ora11g@racnode1 ~]$ srvctl start instance -d racdb -i racdb4
[ora11g@racnode1 ~]$ srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4
[ora11g@racnode1 ~]$




How to check the status of database across cluster nodes:




[ora11g@racnode1 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4



How to stop cluster database:



[ora11g@racnode1 ~]$ srvctl stop database -d racdb
[ora11g@racnode1 ~]$ srvctl status database -d racdb
Instance racdb1 is not running on node racnode1
Instance racdb2 is not running on node racnode2
Instance racdb3 is not running on node racnode3
Instance racdb4 is not running on node racnode4
[ora11g@racnode1 ~]$



How to start cluster database:



[ora11g@racnode1 ~]$ srvctl stop database -d racdb



Note:
- srvctl command can be used from any of the active cluster node to stop/start cluster database.

- These command will be aplicable on other platforms too such as HP-UX, Sun Solaris etc.,


Thanks for reading :)

Syed Zaheer

3 comments:

Sanjib said...

Its very good link about 11gR2 RAC.
Plz upload more about 11gR2 admin commands.....!!!

Seb said...

Hi, you have any idea why this 'nodeapps' option is not documented on page http://docs.oracle.com/cd/E11882_01/server.112/e25494/restart005.htm ??
Thanks.
Seb

Blogger said...

BlueHost is ultimately one of the best hosting company for any hosting services you might need.