Oracle DBA Interview Questions and Answers RAC

===========================================
What is RAC and how is it different from non RAC databases?
RAC stands for Real Application Cluster, you will have n number of instances
running in separate nodes for single database and having shared storage.
RAC is the best solution for high performance and high availably.
Oracle Real Application Clusters (RAC) is a software component we
can add to a high-availability solution that enables users on multiple
machines to access a single database with increased performance. RAC
comprises two or more Oracle database instances running on two or more
clustered machines and accessing a shared storage device via cluster
technology. To support this architecture, the machines that host the database
instances are linked by a high-speed interconnect to form the cluster.
Oracle RAC offers features in the following areas:
Scalability
Availability
Load balancing
Failover
Reliability if one node fails, the database wont fail
Availability nodes can be added or replaced without having to shut down
the database
Scalability more nodes can be added to the cluster as the workload
increases
srvctl
srvctl
srvctl
srvctl
srvctl
srvctl

start instance -d db_name -i "inst_name_list" [-o start_options]

stop instance -d name -i "inst_name_list" [-o stop_options]
stop instance -d orcl -i "orcl3,orcl4" -o immediate
start database -d name [-o start_options]
stop database -d name [-o stop_options]
start database -d orcl -o mount

What is RAC? What is the benefit of RAC over single instance

database?
In Real Application Clusters environments, all nodes concurrently execute
transactions against the same database. Real Application Clusters
coordinates each nodes access to the shared data to provide consistency
and integrity.
Benefits:
Improve response time

Improve throughput
High availability
Transparency
What is Oracle RAC One Node?
Oracle RAC one Node is a single instance running on one node of the
cluster while the 2nd node is in cold standby mode. If the instance fails for
some reason then RAC one node detect it and restart the instance on the
same node or the instance is relocate to the 2nd node in case there is failure
or fault in 1st node. The benefit of this feature is that it provides a cold
failover solution and it automates the instance relocation without any
downtime and does not need a manual intervention. Oracle introduced this
feature with the release of 11gR2 (available with Enterprise Edition).
Real Application Clusters
Oracle RAC is a cluster database with a shared cache architecture that
overcomes the limitations of traditional shared-nothing and shared-disk
approaches to provide a highly scalable and available database solution
for all your business applications. Oracle RAC provides the foundation for
enterprise grid computing.
Oracles Real Application Clusters (RAC) option supports the transparent
deployment of a single database across a cluster of servers, providing fault
tolerance from hardware failures or planned outages. Oracle RAC running on
clusters provides Oracles highest level of capability in terms of availability,
scalability, and low-cost computing.
One DB opened by multiple instances so the db will be Highly
Available if an instance crashes.
Cluster Software. Oracles Clusterware or products like Veritas Volume
Manager are required to provide the cluster support and allow each node to
know which nodes belong to the cluster and are available and with Oracle
Cluterware to know which nodes have failed and to eject then from the
cluster, so that errors on that node can be cleared.

ORACLE RAC ARCHITECTURE 10G AND 11G

Architecture Diagram:

This is a basics architecture of Oracle RAC, We will discuss about new

features/additions in 11g in each sections.
RAC components
Shared Disk System
Oracle cluster ware
Cluster interconnects
Oracle Kernel Components
Shared Disk System
Below are the three major type of shared storage which are using in RAC
Raw volumes: A raw logical volume is an area of physical and logical disk
space that is under the direct control of an application such as database or

partition rather than under the direct control of the operating system or a file
system. This as the only available option in 9i.
Cluster File system: This option is not widely used and here the cluster file
system such as Oracle Cluster file system (OCFS) for MS Windows and Linux
holding the all datafiles of RAC database
Automatic Storage Management (ASM): Oracle recommended storage
option which is optimized for cluster file system for Oracle database files
introduced in Oracle 10g.
Oracle Clusterware
Clusterware is the mandatory piece of software which is required to run
Oracle RAC option,it provides the basic clustering support at OS level and
enables oracle software to run in cluster mode.
Clusterware enables nodes to communicate each other and forms the cluster
that make the nodes work as a single logical server.
Clusterware is managed by cluster ready services (CRS) using the Oracle
Cluster registry (OCR) and voting disk
Cluster Ready Services (CRS)
Oracle clusterware uses CRS for interaction between the OS and the
Database.CRS have five components namely Process Monitor daemon
(OPROCd: obsolete in 11gR2),CRS daemon(CRSd),Oralce Custer
Synchronization Service daemon(OCSSd),Event Volume Manager
daemon(EVMd) and the Oracle Notification Service (ONS)
CRS is installed and run from ORA_CRS_HOME in 10g and GRID_HOME in 11g
Starting and Stopping CRS
crsctl stop crs
# Stops Oracle clusterware
crsctl start crs
# Starts Oracle clusterware
crsctl enable crs # Enables Oracle clusterware
crsctl disable crs # Disables Oracle clusterware
Note: These commands should be used in root user (Superuser)
Oracle Cluster Synchronization Services
OCSSd provides synchronization between the nodes. Failure of OCSSd causes
the machine to reboot to avoid split-brain situation.OCSSd runs as oracle
user.

Event Manager Process

This runs to daemon process EVMd. The daemon process spawns a
permanent child process called evmlogger and generate the events when
things happens.
Oracle Notification Services
Whenever state of cluster resource changes ONS process in each node will
communicate with each other and exchange the high availability information.
Oracle Process Monitor
The oracle process monitor daemon(OPROCd) identifies the potential cluster
node hang and rebooting the hang node.
Oracle Cluster Registry (OCR)
OCR uses the cluster registry to keep the configuration information. This
should be a shared storage and should be accessible to all the nodes in the
cluster simultaneously.This shared storage is known as Oracle cluster registry
(OCR)
Oracle Logical Registry(11gR2)
Similar to OCR, introduces in 11gR2 but it only stores information about the
local node. It is not shared by other nodes of cluster and used by OHASd
while starting or joining a cluster.
Voting Disk
A voting disk is a shared disks that will be accessed by all the member of the
nodes in the cluster. It is used as central reference for all the nodes and
keeps the heartbeat information between the nodes. If any of the node is
unable to ping the voting disk, cluster immediately recogonize the
communication failure and evicts the node from the cluster.
Cluster interconnect
It is the communication path used by the cluster for the synchronization of
resources and it is also used in some cases for transfer of data from one
instance to another. Typically, the interconnect is a network connections that
is dedicated to the server nodes of a cluster (thus is sometimes referred as
private interconnect)

Single Client Access Name (SCAN)-11gR2

SCAN is a single network name that resolves in three different IPs registered
in DNS or GNS.
Prior the 11g R2 if we add or remove a node from cluster it was required to
changes the connection setting in client. By configuring the connection using
SCAN name this problem is eliminated.
Oracle Kernel Components
Set of additional background process in each instance is known as oracle
kernel components in RAC environment.Since buffer and shared pool became
global in RAC , special handling is required to manage the resources to avoid
conflicts and corruption.Additional background process (for RAC) and single
instance background process works together and achieved this.
Global Cache and Global Enqueue Services
In Oracle RAC each instance will have its own cache but it is required for an
instance to access the data blocks currently residing in another instance
cache.This management and data sharing is done by Global Cache services
(GCS). Blocks other than data such as locks, enqueue details and shared
across the instances are known as Global Enqueue Services (GES).
Global Resource Directory
In a cluster group, then centre repository formed by all the resources is
known as Global Resource Directory (GRD).GRD is managed by GCS and GES.
Oracle RAC Background Processes
LMS
Global Cache Service Process
LMON Global Enqueue Service Monitor
LMD
Global Enqueue service Daemon
LCK0
Instance Enqueue Process
LMS: Global Cache Services Process
LMS- Lock Manager Server Process is used in Cache Fusion.It enebles
consistent copies of blocks to be transfered from a holding instances buffer
cache to a requesting instance bufer cache with out a disk write under
certian conditions.
It rollbacks any uncommitted transactions for any blocks that are being
requested for a consistent read by the remote instance.

LMON:Global Enqueue Services Monitor

LMON-Lock Monitor Process is responsible to manage Global Enqueue
Services (GES).It maintain consistency of GCS memory in case of any process
death. LMON is also responsible for the cluster reconfiguration when an
instance joins or leaves the cluster. It also check for the instance death and
listens for local manages.
LMD: Global Enqueue Services Daemon
LMD-Lock Manager Daemon process manages Enqueue manager service
requests for GCS. It also handles deadlock detection and remote resource
requests.
LCK0:Instance Enqueue Process
LCK-Lock Process manages instance resource requests and cross-instance
call operations for shared resources.
DIAG Process
Separate process using for diagnostic purpose, PMON will start incase this
process dies.DIAG process will not affect any other background process.

RAC startup and shutdown procedure:

---------------------------------------------------

How to Shutdown Oracle Real Application Clusters Database?

1.
2.
3.
4.
5.

Shutdown Oracle Home process accessing database.

Shutdown RAC Database Instances on all nodes.
Shutdown All ASM instances from all nodes.
Shutdown Node applications running on nodes.
Shut down the Oracle Cluster ware or CRS.

1. Shutdown Oracle Home process accessing database:

[grid@node1 bin]$ srvctl stop listener -n node1
[grid@node1 bin]$ srvctl status listener -n node1
Listener LISTENER is enabled on node(s): node1
Listener LISTENER is not running on node(s): node1
2. Shutdown RAC Database Instances on all nodes:

[oracle@node2 ~]$
Instance RACDB1 is
Instance RACDB2 is
[oracle@node2 ~]$

srvctl status database -d RACDB

running on node node1
running on node node2
srvctl stop database -d RACDB

[oracle@node2 ~]$ srvctl status database -d RACDB

Instance RACDB1 is not running on node node1
Instance RACDB2 is not running on node node2
***We just need to execute one command from any one of the server
having database and it will stop all database instances on all
servers.
3. Shutdown All ASM instances from all nodes:
[grid@node2 oracle]# srvctl stop asm -n node1 -f
[grid@node2 oracle]# srvctl stop asm -n node2 -f
[grid@node2 oracle]# srvctl status asm -n node1
ASM is not running on node1
[grid@node2 oracle]# srvctl status asm -n node2
ASM is not running on node2
***Sometimes, Database administrator face some issues in stopping
ASM instance, In that case use "-f" option to forcefully shutdown
ASM instances.
4. Shutdown Node applications running on nodes:
[grid@node2 oracle]# srvctl stop nodeapps -n node1 -f
[grid@node2 oracle]# srvctl status nodeapps -n node1
VIP node1-vip is enabled
VIP node1-vip is running on node: node1
Network is enabled
Network is running on node: node1
GSD is disabled
GSD is not running on node: node1
ONS is enabled
ONS daemon is running on node: node1

***Repeat same command for all nodes one by one. If you face any
issue in stopping node applications use "-f" as force option to stop
applications.
5. Shut down the Oracle Clusterware or CRS:
[root@node1 bin]# crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
*************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@node1 bin]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed
resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources
on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node1'
------------------------------------------------------------------------------------------------------------------------------------------------CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources
on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@node1 bin]# crsctl check cluster -all

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Check failed, or completed with errors.
How to Start Oracle Real Application Clusters Database ?
1.
2.
3.
4.
5.

Start
Start
Start
Start
Start

Oracle Cluster ware or CRS.

Node applications running on nodes.
All ASM instances from all nodes.
RAC Database Instances on all nodes.
Oracle Home process accessing database.

1. Start Oracle Clusterware or CRS:

[root@node1 bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started
[root@node2 bin]# crsctl check cluster -all
**************************************************************
node1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
***Here, DBA can see "CRS-4639: Could not contact Oracle High
Availability Services" or "CRS-4535: Cannot communicate with
Cluster Ready Services" messages. Wait 5 minutes and then again
check with "crsctl check cluster -all" command. This time Database
administrator will get "CRS-4537: Cluster Ready Services is online".
*****If still same issue DBA can start ora.crsd process to resolve this
issue. Below is the command
[root@node1 bin]# crsctl start res ora.crsd -init
[root@node1 bin]# crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
2. Start Node applications running on nodes:
[grid@node1 bin]$ srvctl start nodeapps -n node1
[grid@node1 bin]$ srvctl status nodeapps -n node1
VIP node1-vip is enabled
VIP node1-vip is running on node: node1
Network is enabled
Network is running on node: node1
GSD is disabled
GSD is not running on node: node1
ONS is enabled
ONS daemon is running on node: node1
***DBA has to execute this command for each node to start Real
Application Clusters Cluster database.
3. Start All ASM instances from all nodes:
[grid@node1 bin]$ srvctl start asm -n node1
[grid@node1 bin]$ srvctl status asm -n node1
ASM is running on node1
DBA has to start ASM instance on all database nodes.
4. Start RAC Database Instances on all nodes:
[grid@node1 bin]$ srvctl start database -d RACDB
[grid@node1 bin]$ srvctl status database -d RACDB
Instance RACDB1 is running on node node1
Instance RACDB2 is running on node node2

5. Start Oracle Home process accessing database:

[grid@node1 bin]$ srvctl start listener -n node1

[grid@node1 bin]$ srvctl status listener -n node1
Listener LISTENER is enabled on node(s): node1
Listener LISTENER is running on node(s): node1

About Oracle Real Application Clusters Software Components

Oracle RAC databases have two or more database instances that each
contain memory structures and background processes. An Oracle RAC
database has the same processes and memory structures as a singleinstance Oracle database as well as additional process and memory
structures that are specific to Oracle RAC. Any one instance's database view
is nearly identical to any other instance's view in the same Oracle RAC
database; the view is a single system image of the environment.
Each instance has a buffer cache in its System Global Area (SGA). Using
Cache Fusion, Oracle RAC environments logically combine each instance's
buffer cache to enable the instances to process data as if the data resided on
a logically combined, single cache.
Note:
The SGA size requirements for Oracle RAC are greater than the SGA
requirements for single-instance Oracle databases due to Cache
Fusion.
To ensure that each Oracle RAC database instance obtains the block that it
needs to satisfy a query or transaction as fast as possible, and to ensure
data integrity, Oracle RAC instances use two processes, the Global Cache
Service (GCS) and the Global Enqueue Service (GES). The GCS and GES
maintain records of the statuses of each data file and each cached block
using a Global Resource Directory (GRD). The GRD contents are distributed
across all of the active instances, which increases the size of the SGA for an
Oracle RAC instance.
After one instance caches data, any other instance within the same cluster
database can acquire a block image from another instance in the same
database faster than by reading the block from disk. Therefore, Cache Fusion
moves current blocks between instances rather than re-reading the blocks
from disk. When a consistent block is needed or a changed block is required
on another instance, Cache Fusion transfers the block image directly
between the affected instances. Oracle RAC uses the private interconnect for
interinstance communication and block transfers. The GES Monitor and the

Instance Enqueue Process manages access to Cache Fusion resources and

enqueue recovery processing.
About Oracle Real Application Clusters Background Processes
The Oracle RAC processes and their identifiers are as follows:
ACMSAtomic Controlfile to Memory Service (ACMS) from 11g
In an Oracle RAC environment, the atomic control file to memory service
(ACMS) per-instance process is an agent that contributes to ensuring a
distributed SGA memory update is either globally committed on success or
globally aborted in the event of a failure.
GTX0-jGlobal Transaction Process from 11g
The GTX0-j process provides transparent support for XA global transactions
in a RAC environment. The database auto tunes the number of these
processes based on the workload of XA global transactions.
LMONGlobal Enqueue Service Monitor
The LMON process monitors global enqueues and resources across the
cluster and performs global enqueue recovery operations.
LMDGlobal Enqueue Service Daemon
The LMD process manages incoming remote resource requests within each
instance.
LMSGlobal Cache Service Process
The LMS process maintains records of the data file statuses and each cached
block by recording information in a Global Resource Directory (GRD). The
LMS process also controls the flow of messages to remote instances and
manages global data block access and transmits block images between the
buffer caches of different instances. This processing is part of the Cache
Fusion feature.
LCK0Instance Enqueue Process
The LCK0 process manages non-Cache Fusion resource requests such as
library and row cache requests.
RMSnOracle RAC Management Processes (RMSn)

The RMSn processes perform manageability tasks for Oracle RAC. Tasks
accomplished by an RMSn process include creation of resources related
Oracle RAC when new instances are added to the clusters.
RSMNRemote Slave Monitor manages background slave process creation
and communication on remote instances. These background slave processes
perform tasks on behalf of a coordinating process running in another
instance.
What are Oracle database background processes specific to RAC?
LMSGlobal Cache Service Process
LMDGlobal Enqueue Service Daemon
LMONGlobal Enqueue Service Monitor
LCK0Instance Enqueue Process
Oracle RAC instances use two processes, the Global Cache Service (GCS) and
the Global Enqueue Service (GES). The GCS and GES maintain records of the
statuses of each data file and each cached block using a Global Resource
Directory (GRD). The GRD contents are distributed across all of the active
instances.