Oracle RAC Installation

Prerequisites

 

Hardware

 

At least 512MB of RAM on each mode. Recommended having 2GB min, 4Gb maximum (twice the physical RAM size).

Disk space

To support RAC your database must be stored on disks that are shared by both nodes on your computer. Any shared disks supported by your hardware vendor other than Network Attached Storage (NAS) can be used. The shared disks must be attached to both nodes in your cluster and both nodes must be able to read and write to them. For performance reasons, you should use at least two shared disks for your database files.

 

On LINUX, install kernel-unsupported-2.4.21-4.EL.i686.rpm from RedHAt Disk3 on each node.

On LINUX, add the following lines to the /etc/modules.conf on each node:

Options sbp2 sbp2_exclusive_login=0 (both nodes can access the disk simultaneously)

Post-install sbp2 insmod sd_mod

Post-remove sbp2 rmmod sd_mod

After the reboot run the following commands to detect the cards and the drive

Modprobe sd_mod

Modprobe scsi_mod

Modprobe ohci1394

Echo “scsi add-single-device 0 0 0 0” > /proc/scsi/scsi

 

 

Requirements for 10g RAC installation and RAC configuration are

 

Requirement	Free Space Needed	Location
Ancillary Oracle files	100MB on both nodes	Any local system drive
Temporary space	50MB	Any local system drive
Cluster ready services	500MB on both nodes	Any local system drive must be the same named drive on both nodes
Database software	1GB on both nodes	Any local system drive, but must the same named on both nodes
Database files	7GB	Shared disks

 

Network

 

The two nodes in the cluster must be able to communicate with each other and with external clients using TCP/IP protocol. Communication between the clients and nodes in the cluster is across the public network. Both nodes need a network adapter configured for the private network.

To enable availability and failover, a virtual IP address is also required for both nodes. A VIP address can be moved between the nodes in case of failure. CRS manages the VIP address for you.

To support a virtual IP address, both nodes require an unused IP address that is compatible with the public network’s subnet and netmask.  The virtual IP address and host name should also be registered in the domain name system. For communication between the instances running on two nodes a private network is required. This private network connects only the nodes in the cluster and cannot be accessed outside the cluster. Both nodes need a separate network adapter configured for this private network.

Public versus private network

Both nodes must meet the following public and private network requirements.

1. Support 2 network adapters – one for public interface and client connections, the other one is private for interconnect.
2. Support interface name restrictions

-         Names are case- sensitive

-         Names must not contain any multibyte language characters

-         Public and private network interface names must be differentiated from each other.

-         Names for the same interface should be the same on both nodes.

-         Public and private IP addresses must be on different subnets.

3. An IP address and host name registered in the domain  name system (DNS) for the public network interface.
4. An IP address and optionally a private hostname, for each private network interface.

Note: Oracle recommends that you use private network IP address for these interfaces. You can use drivers\etc\hosts on both nodes to associate private host names with private IP address.

 

Host Name	Type	IP Address	Registered In
iwinrca01.us.oracle.com	Public	139.185.150.201	DNS
iwinrca02.us.oracle.com	Public	139.185.150.202	DNS
viprca01.us.oracle.com	Virtual	139.185.150.54	DNS
viprca02.us.oracle.com	Virtual	139.185.150.55	DNS
iwinrca01-node	Private	10.10.10.2	Hosts file
iwinrca02-node	Private	10.10.10.3	Hosts file

 

To configure IP addresses associated with your cluster nodes, perform the following steps:

1. IPCONFIG on all nodes.
2. Make sure that interface names match that on node one.
3. Confirm that the private IP addresses are listed in the SYSTEMROOT…etc\hosts file.
4. Confirm that two nodes can actually communicate – ping both private and public IPs on both nodes.

 

Configure the Disk System

 

1. Disable write caching

Start, settings, control panel, administrative tools, computer management, device manager, DiskDrives, right click properties, disk properties, uncheck Write Cache enabled. Do this for all disk drives.

2. Prepare disks for Cluster Ready Services

Cluster ready Services (CRS) provides management of cluster activities. CRS requires that the following key files be located in logical drives on shared disks:

Oracle Cluster Registry

Voting disk

1. Start, Settings, Control Panel, Administrative Tools, Computer management, expand storage and disk management.
2. Right click in the free space area on disk1 and select Create Logical Drive.
3. Enter 100MB for the size of the first logical partition. This is the size you want for the Oracle Cluster Registry. Click next. Select Do not assign a drive letter or drive path option and click Next. Choose do not format this partition and choose next, then Finish.
4. Repeat the previous steps to create a second logical drive of 20MB for the Voting disk.
5. Check the two nodes in the cluster to make sure the partitions are visible on both and none of the Oracle partitions have drive letters assigned.

 

 

Prepare disks for database storage

 

You must configure disk storage for use with Automatic Storage Management – (ASM). ASM storage consists of one or more disk groups, each of which spans multiple disks. To prepare the shared disks you identified in the Disk Space prerequisite, you need to create two more logical drives, each on a different disk, for a total of at least 7GB.

In this example, only one disk is available, so two additional logical drives were created on Disk 1. All of the partitions you create for your ASM disk groups must have at least 7GB, be equal in size, and include as much of the free space as possible.

 

 

Installing CRS – Cluster Ready Services

 

1. Run  setup.exe from the Oracle Cluster Ready Services CD-ROM
2. Click Next in Welcome screen
3. Specify file location for Oracle files.
4. On the cluster configuration page enter a cluster name, your public and private node name for both nodes.
5. On the Specify Network Interface Usage page, the Universal installer displays a list of cluster wide interfaces. Select public for the first Local area connection interface and Private for the Local Area Connection 2 Interface and click next.
6. On the Select Disk Formatting options select DO not format any logical drives and click next.
7. On the Disk configuration – Oracle Cluster registry (OCR), select the partition’s disk number to hold the OCR (100MB) and click Next.
8. On the Disk configuration – Voting Disk page, select the partition’s disk number to hold the voting disk (20 MB) and click next.
9. The Configuration Assistants appear.
10. End of installation.

 

Stamping the Logical Drives for ASM

 

To enable disk discovery during the database installation, the logical drives used to store your database files must be stamped with an ASM header using asmtoolg. All disk names created by the tool begin with the prefix ORCLDISK for identification purposes. To stamp the logical drives for ASM perform the following tasks:

1. Run <CRS_HOME>\BIN\asmtoolg.exe.
2. Accept the default add or change label and click Next.
3. On stamp disks page select a disk to stamp and optionally modify the prefix. Click Next, click Finish. Repeat the steps above for the second logical disk.

 

 

Installing Oracle database 10g RAC

 

1. Run setup.exe from the Oracle CD-ROM.
2. Choose file location
3. On the Specify Hardware Cluster installation page, the cluster installation mode is selected by default. In addition, the local mode is always selected. Select additional nodes to include I the cluster and click next.
4. Select standard edition, click Next.
5. On the select database configuration page accept the default options to create a general purpose database and click Next, Create database with sample Schema.
6. On th Select database management option select Automatic Space management (ASM) and click next.
7. On the Specify backup and Recovery Options choose Enable Automated Backups.
8. On the Configure Automatic Storage management Option select the check box for each disk that you prepared for your ASM disk groups. The Universal Installer lists only the partitions you labeled with asmtoolg. Then click next.
9. Specify schema passwords.
10. When the installation completes, the Welcome page for the virtual IP configuration assistant appears. Click next.
11.  On the Network Interfaces page, select the network interface card (NIC) to which you want to assign your VIP address. Then Click next. Make sure your public NIC is selected for this purpose. If you select your private NIC, your cluster will not function properly.
12.  On the Virtual IPs for cluster nodes page enter the virtual IP you identified in the network prerequisites section and click next.
13. Click Finish in the Summary screen.
14. After VIPCA the NET configuration assistant will run, then the Database configuration assistant.
15. After that: Starting cluster database “orcl” and its instances “orcl1, orcl2” in progress.
16. End of installation.

 

 

Verifying a valid Path name for Both Nodes

 

Select Environment Variables, path. Make sure the value is set to <ORACLE_HOME>\bin. If there is no value – add it. Repeat the step for the other node.

 

Reviewing your installation in Oracle EM database control.

You should see a cluster with 2 instances.

 

 

 

 

RAC Specific Parameters

 

Instance name vs database name

As a rule you want the instance name begin with the database name – db_name1, db_name2

 

Spfile in the RAC world

 

The only way to change the spfile is by using ALTER SYSTEM. In a RAC environment all instances should share the same parameter file. If a parameter has to be unique for an instance, it must be prefaced with the instance name <sid_name>.<parameter_name>. For parameters that apply to all instances the instance name is replaced by an asterisk *.

Grid2.instance_number=2

Grid1.instance_number=1

*.java_pool_size=50331648

*.job_queue_processes=2

 

Specifying the SID when modifying parameters

 

The SID option tells Oracle which sid the change applies to.

SQL> ALTER SYSTEM SET db_cache_size=500m SCOPE=SPFILE SID=’grid1’;

Only grid1 will have db cache of 500M.

 

RAC Specific Parameters

 

CLUSTER_DATABASE=TRUE – defines instance as a part of a cluster. It determines that controlfile will be mounted in a shared mode (other instances can mount it). This value should be set to false if upgrading or applying a patch.

 

CLUSTER_INTERCONNECTS – specifies network that is used for the IPC traffic to the cluster. You must do this for every sid:

ALTER SYSTEm SET cluster_interconnects=’10.1.1.1’ SCOPE=SPFILE SID=’grid1’;

 

ACTIVE_INSTANCE_COUNT applies only to two node cluster. If it is set to a value of 1, in a 2 node cluster it will consider that the first instance to start will be the primary instance. The second instance will secondary and will only be active if the first instance goes down.

 

MAX_COMMIT_PROPAGATION_DELAY – defines the maximum amount of time that SCN is held in the SGA of the local system before it is refreshed by the LGWR process. The default is 700 (7 seconds). The downside of setting this parameter to 0 is that it will be a slight performance impact as all inserts and updates propagate immediately.

 

Additional Background Processes in RAC

DIAG – the diagnosibility daemon is responsible for capturing information on process failures in RAC

 

LCK – lock process manages requests that are not cache-fusion requests, such as row cache requests and library cache requests.

 

LMD – lock manager daemon process or global enqueue service daemon as it manages global resources requests. It is also responsible for deadlock detection.

 

LMON – lock monitor process is responsible for the lock reconfiguration when the instance joins or leaves the cluster.

 

LMS – lock manager server process is in charge of shipping the blocks between instances for cache fusion requests. An instance might dynamically configure up to 10 LMS processes.

 

GV$ views – same as V$, but for Global.

 

 

Beginning with version 7.3.3.0.1 Oracle began providing a clustering layer for Windows., with 9i – on Linux as well. Starting with 10g, Oracle introduced CRS (Cluster-Ready Services) that can connect together nodes on any supported OS (all nodes must be on the same OS, though).

 

CRS - Cluster-Ready Services Architecture

CRS consists of 3 major components – daemons - running out of inittab on UNIX or Services on Windows.

-         ocssd (cluster synchronization services daemon)

-         crsd (main engine of availability of resources)

-         evmd (event logger daemon)

Of these 3 components, ocssd and evmd run as user oracle, while crsd runs as root.

 

CSS: Cluster Synchronization Services

CSS is the foundation of interprocess in a cluster environment. It provides information on which nodes and instances are parts of the cluster, rudimaentary locking mechanism (mostly handled by RDBMS, though), maintains a heartbeat between the nodes.

 

CRSD

The CRSD daemon is primarily responsible for maintaining the availability of application resources – services, starting and stopping them, moving them to other nodes, and maintaining the services profiles in Oracle Cluster Registry (OCR). In addition it is responsible for caching of the OCR for faster access and backing it up.

 

Virtual IPs, or VIPs

Each node will have not only its own statically assigned IP, but also have a virtual IP assigned to the node. The listener on the node will be listening on the single VIP, and the client connections are meant to come in on the VIP. Should the node fail, the virtual IP will actually fail over and come online on one of the other nodes in the cluster – rapid connect-time failover.

 

CRS Installation

CRS is provided on a separate CD-ROM from Oracle RDBMS. It needs to be installed before RDBMS and goes into its own home CRS_HOME. For this reason, if you may plan to use RAC in the future, you may want to install CRS first, then regular Oracle install, no RAC option. By doing this, you will enable the ocssd daemon running out of the CRS home, and this daemon can then be used for ASM (Automatic Storage Manager) by all other Oracle 10g installations – local or RAC.

 

OS Configuration for CRS

 

Storage Requirements for CRS/RAC

CRS gives you 3 possible configurations to be used for storage: to use either RAW devices or Automatic Storage management for shared files required by CRS and RAC. In addition, Oracle offers its own cluster file system (OCFS) on LINUX and Windows. A fourth option on Linux is to use NFS mounts via a certified network attached storage device (Network Appliance) for storage of the shared database and CRS files.

 

Shared Everything Disk Storage

Shared everything means that they all involve disks accessible from multiple nodes simultaneously. Shared nothing means that only one node can access a disk at a time. Oracle requires that all controlfiles, all online redo logs, and all datafiles be stored on the shared drive. In addition, archived redo log files, system parameter file and flashback logs can be shared on the shared drive.

 

Straight Raw Devices

If using straight RAW devices for all files, you are essentially forgoing a file system altogether. Since there is no file system, you will have to create a separate partition for every file you create for the database. As such, it is not practical to store archived logs on RAW devices. They do allow for faster I/O, though.

 

Automatic Storage Management – Designed for RAC

ASM allows the use of block or RAW devices that are managed completely by Oracle. All file management, including backups, is done from within database. While archived logs and flashback logs can be stored on ASM disk groups, the Oracle binaries cannot, so it is still required to either have a cluster file system for the Oracle install or install to the private drives on each node.

 

Cluster File System

Allows directory structure simplicity which can be used for all database files. It allows a directory structure – multiple files can be stored on the same single shared partition/disk. On some platforms CFS can be used as a shared location of the Oracle binaries.

 

NFS Mounts

On LINUX, it is also possible to have the shared device for the datafiles be on an NFS mount, using a certified network attached storage vendor. This requires that all nodes have network access to the device and have the same NFS mount point. The advantage that it does also allow file system management, easing the complexity. The disadvantage is that network latency can slow down performance. While NFS mount points should not be used for the Oracle home NFS mounts as a location for archived logs via flashback recovery area.

 

Networking Requirements for CRS and RAC

 

Two network cards in each node at a minimum (one card for client communication, one for cluster communication). With this configuration in mind you have to first configure the /etc/hosts file on each node to have a private name for both private and public nodes: node1, node1_private, node2, node2_private. Although on some platform it is possible to have a simple crossover cable, it is not recommended because of the media sensing behavior of the OS. On Windows, foe example, if one node is powered down, the card for the surviving node is disabled because there is no more activity being sensed across interconnect. This can lead to errors on the surviving node.

The best solution is to have a dedicated switch between the nodes, which means there is a constant activity on the card (coming from the switch) even if the other node is down. Of course, this also allows expansion of your cluster to more than just two nodes. The other consideration is that the switch and the cards for the interconnect should be capable of handling of all of the cache fusion traffic across the interconnect – at least 1GB Ethernet or faster.

 

Kernel Parameters

 

For Red Hat LINUX 3.0 kernel changes can be made by making modifications to the /etc/sysctl.conf file.

Add the following:

 

Kernel.shmall = 2097152

Kernel.shmax = 536870912

Kernel.shmni = 4096

Kernel.sem = 250 32000 100 128

Fs.file-max = 65536

Net.ip4.ip_local_port_range= 1024 6500

 

 For Kernel.shmax the value of 536870912 is equal to half of the physical RAM installed one each node.

 

OCR and Voting Disk Requirements

 

As a part of CRS install, you will be prompted to provide a separate location for the Oracle Cluster Registry (OCR) and the voting disk used by CRS. These are required to be stored on a shared disk, as both nodes have to have access to them. The OCR is essentially a metadata database for your cluster – keeping track of resources and which node they should be located on. The voting disk will be used for resolving split-brain scenarios – in case of network failures through the interconnect.

 

 

Oracle RAC Installation on Windows

 

Use raw partitions

 

Because all nodes are accessing drives concurrently, you cannot rely on a normal on normal NTFS file system to maintain access to these drives and avoid disk or data corruption. Instead, this is accomplished by leaving these partitions unformatted, or raw with disk cache disabled. This allows the DLM (Distributed Lock manager) within RDBMS to control access to the data blocks, ensuring that only one node is writing a given block at any given time.

Since there is no file system on a raw partition, there can be only one file per partition. All database files must be on raw device – including data files, redo log, controlfiles and even SPFILE. An exception to this is archived logs, which are written to a file system. Once all disks are partitioned and, Oracle accesses the raw partition as a symbolic link.

 

RAC cluster software

 

Voting disk

Similar to the quorum disk used by MCS, the RAC cluster software requires that one of the shared drives be configured as a voting disk. This is needed to resolve conflicts between nodes. The voting disk contains information about Oracle homes and instances on each node. This information is read by DBCA, OEM or other Oracle tools.

 

Preparing for the Cluster Installation

 

1. Ensure that interconnect is configured correctly
2. Partition the disks according to how they will be used, keeping in mind that each file will require its own partition.
3. Finally, you must define the links to these raw partitions. This step is done using Oracle Object Link Manager or Cluster Setup Wizard.

 

1. Configuring the interconnect

You will need 2 network cards on each node. Ensure that you that public network card is bound first, and the card with the private IP is bound last. The private interconnect should use at least a gigabit connection. Also, since the private network is not connected to a DNS server, you have to define a network name for the private IP in the HOSTS file on each node.

 

127.0.0.1 localhost

10.10.10.1 RMNTOP1.SAN

10.10.10.2 RMNTOPS2.SAN

 

2. Configuring the raw partitions

First, determine how many shared devices you have available. Next, determine how many of them you can use for RAC.

 

3. Link names and partitions required by the database assistant

An example would be where DBCA expects 17 partitions.

1 - SPFILE

2 – control files

4 - online redo logs (2 groups)

2 – undo tablespaces

8 – data file partitions (system, temporary, drsys, cwmlite, example, users, index tools)

1 - voting disk to allow the cluster ownership of the disks if the interconnect fails.

 

Mapping link names to devices

Go to Disk Manager, create the partitions as basic partitions, not dynamic disks (unsupported). Do not cerate a primary partition, so you are limited to the total number of partitions.

Creating Logical drives

Once you have created an extended partition, go back and cerate a logical drive, do not assign a drive letter or path, do not format or enable disk cache.

 

To create the object links, double click on GUIOracleOBJManager.exe, choose a new link, enter the link name beginning with the global database name, choose commit. Options, export links.

 

Additional utilities for managing raw partitions – crlogdr (create logical drives), logparformat formats raw partitions.

 

Running RAC Clustercheck

Clustercheck is in the \preinstall_rac\clustercheck directory.  It has to be run prior to installation of RAC.

 

Defining public and private names for clustercheck

You will be asked for cluster nodes and the public and names for both nodes. The host name for node 1 has to be the host name from which you are running the install. Clustercheck will ask for private and public names.

 

Check clustercheck log files

During the clustercheck  a service InfoGatherer will be created on each node. The utility creates a log in the TEMP directory called something like OraInfoCoord.log. If it has an ORACLE CLUSTER CHECK WAS SUCCESFULL in it, you can proceed.

 

Installing the Cluster Software and Oracle RDBMS

 

Creating the Cluster

Defining links in cluster setup – example: RACDB_REDO01, disk 2, partition 1, size 200M.

 

Defining the network for interconnect in the Network Configuration – public name – RMNTOPS1, private name – RMNTOPS1.SAN.

 

Installing the Oracle software

 

The database name has to match the symbolic links you specified – RACDB. If installing on more than 2 node RAC, obtain the patch BUG#2031489 before the install. You can install either a single instance or a true RAC database. After the installation, make sure that the PATHs are the same on all nodes – reboot if necessary. Query V$ACTIVE _INSTANCES to

 

Maintaining a RAC database

 

SPFILE in RAC

Make sure the CLUSTER_DATABASE=TRUE, the rest is in the other pages

 

System-managed UNDO and/or rollback tablespaces in RAC

Each instance must have its own undo tablespace.

 

Redo log groups

Each instance must maintain a set of two redo log groups, minimum 2 per instance. If you want to add or remove groups you have to disable the thread:

SQL> ALTER DATABASE DISABLE THREAD 2;

Even though the logs belong to a particular instance, they must exist on raw devices, so either instance can read them remotely.

 

Archiving with multiple threads of redo

All archived redo logs must go to a file system, so it has to be on a private drive. The thread number must always be in the LOG_ARCHIVE_FORMAT ensuring that each archived log file will always be created with a unique name.

 

Adding additional instances – use DBCA

 

Adding datafiles and tablespaces

You need to know in advance the size of the partition for the datafile and create it as a logica dirve using Disk Management. You can delete the drive letter from the additional nodes using letterdelete. Run Oracle Object Link manager and define the link name for the partition you have created. Next, add the datafile or partition:

SQL> ALTER TABLESPACE users ADD DATAFILE ‘\\.\RACDB_JUNK’ SIZE 100M;

Remember, the size you specify should be 1% smaller than the partition size.

 

Using the ocopy to backup files on raw partitions

Since the files cannot be copied from raw partition and tape devices do not copy them directly from raw devices, it is common to use RMAN. You must first to ocopy to disk from raw partition, and then to tape.

 

 

 

Practice: Real Application Cluster SPFILE

 

In this practice you will start the gsd process and setup the spfile for the RAC environment.

 

 

Assumptions

 

You will need a clustered hardware environment and RAC installed to run the exercises.

 

 

Instructions

 

1.      1.      Start the Global Services Daemon (GSD) on your primary and secondary nodes. Do this by entering the command gsd at the operating system prompt (you can host out of your SQL*Plus session if it is still active). Note that if GSD is already active on a node, you receive the following message: “An Oracle Real Application Clusters Manageability daemon is already running on this node, please stop it before retrying.”

 

SQL> HOST gsd   Successfully started the daemon on the local node.

 

 

2.      2.      Use the SRVCTL utility to add your database to the repository configuration. Query the name column in the V$DATABASE dynamic performance table for the database name to use with the –p option in your srvctl add db command and use the value $ORACLE_HOME for the –o option.

 

SQL> SELECT name   2  FROM v$database;   NAME --------- U1   SQL> HOST srvctl add db -p U1 -o $ORACLE_HOME Successful addition of cluster database: U1

 

 

1.      3.      Use the srvctl add instance command to add each of your instances to the repository configuration. You will need the database name you used in step 2 (name from V$DATABASE) for the –p option, the instance_name value from V$INSTANCE for the –i option, and node name returned by the operating system hostname command for the –n option.

 

SQL> SELECT name   2  FROM v$database;   NAME --------- U1   SQL> SELECT instance_name   2  FROM v$instance;   INSTANCE_NAME ---------------- U1N1   SQL> HOST hostname git-opssun1   SQL> HOST srvctl add instance -p U1 -i U1N1 -n git-opssun1 Instance successfully added to node: git-opssun1 . . . SQL> HOST srvctl add instance -p U1 -i U1N2 -n git-opssun2 Instance successfully added to node: git-opssun2

 

 

2.      4.      Use the srvctl config –p command with your database name to confirm the repository entries you have added.

 

SQL> HOST srvctl config -p U1 git-opssun1 U1N1 git-opssun2 U1N2

 

 

5.  On your primary node, start a SQL*Plus session in privileged mode and use this session to create a server parameter file with the CREATE SPFILE command. You will need to identify the link for the SPFILE in your $HOME/DATA subdirectory and the initialization parameter file you used to start the instance on this node. You will need to include the full path names for these files although you can substitute $HOME for leading part of each name. Log out of your SQL*Plus session when you are finished.

 

$ sqlplus /nolog SQL*Plus: – Production (c) Copyright 2000 Oracle Corporation.  All rights reserved.   CONNECT sys/oracle AS SYSDBA Connected to an idle instance. SQL> CREATE SPFILE='$HOME/DATA/rac_spfile'   2  FROM PFILE='$HOME/ADMIN/PFILE/initU1N1.ora';   File created.