This is a tutorial and a step by step installation guide for CentOS 5.6 cluster with Oracle 11gR2
http://dl.dropbox.com/u/30415275/66821384-Oracle-Cluster-on-CentOS-Using-CentOS-Cluster-Ware.pdfThis document implements Oracle as a failover service in a non-RAC environment using CentOS clusterware.
This document assumes that you have two nodes with shared storage. If you do not have shared storage
you can implement it using Openfiler, a guide to do that is available at:
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_10.shtml#Configure iSCSI Volumes using Openfiler
The document which details the non-RAC CentOS clusterware installation of Oracle 11gR2 on CentOS5.6 is available at: http://dl.dropbox.com/u/30415275/66821384-Oracle-Cluster-on-CentOS-Using-CentOS-Cluster-Ware.pdf
Introduction
This document will cover the setup/configuration of CentOS cluster and Oracle 11g database as a cluster resource.
Download Software and assumptions
- CentOS 5.6 64bit
- Oracle Database 11g R1 binaries
- You should have internet access for yum install
- ILO cards should have been configured for fencing
- Both CentOS nodes should communicate with each
other with: 1 public interface (bonding is suggested)
1 private interconnect (bonding is suggested)
SAN storage LUNs should be accessible to both nodes, as cluster storage.
Modify /etc/hosts in both nodes
/etc/hosts file in each server should be modified to point to ilos, private interfaces and public interfaces. See the example bellow.
[root@db2 ~]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
#::1 localhost6.localdomain6 localhost6 ## comment this line
192.168.0.70 db1.dyndns.org ## This is the public hostname for db1
192.168.2.70 db1 ## This is the private hostname for db1
192.168.0.80 db2.dyndns.org ## This is the public hostname for db2
192.168.2.50 db2 ## This is the private hostname for db2
# Cluster VIP for Oracle
192.168.0.90 oraprd.dyndns.org ##Cluster Virtual Host
192.168.0.151 ilodb1 ## Ilo1
192.168.0.152 ilodb2 ##Ilo 2
[root@db2 ~]#
Discovering the needed storage
Perform: fdisk –l to check the attached disks. Your storage admin should have assigned those disks to both nodes. A reboot of the servers maybe necessary to see the new disks.
lsscsi rpm installation in both nodes, must be performed.
Check lsscsi output in both nodes. Should display the same disks.
Before the Reboot, you may try the following command, to discover the disks LUNs online (should run in both nodes)
echo "- - -" > /sys/class/scsi_host/host3/scan
Where host3 can be replaced by the output of the command:
lsscsi –H (this command will display all fiber cards connected to your server)
[root@db11 ~]# lsscsi -H
[0] <NULL> à
This means that the fiber card has the instance 0, so it would be host0
You must have already opened the /var/log/messages in each node, from another putty session, because you will probably see the attached new devices there, the moment they are discovered by the previous command.
Create the disk layout (LVM) – This is to be done only in one node
Create 4 pvs. This configuration is enough for a simple Oracle database installation, with no heavy usage.
If a device, has previous partitions because of previous usage (e.g Solaris), then before the pv creation, you need to delete those partitions. Use fdisk command to connect to the device.
Create the volume Groups
Create vg_orabin, vg_oradata, vg_redo1, vg_redo2
Create the Logical Volumes
Create lv_orabin, lv_oradata, lv_redo1, lv_redo2
Create the file systems – If a logical volume is inactive, you cannot create the filesystem, so then you must try to activate with: lvchange –ay lv_name
Reboot the other node, to read all LVM changes (this is needed, according to my experience)
[root@db2 ~]# shutdown -ry now
After the reboot:
Install all cluster and cluster storage rpms (IN BOTH
NODES)
This includes cman, rgmanager, clvmd, ricci, luci, system-config-cluster and lvm2-cluster
Start the cluster config gui (you may use luci web interface to build the cluster, you have to start ricci and luci services first)
Click on Create New Configuration
Name the cluster, do not use quorum yet, click ok
Click on Add a Cluster Node, put the private hostname (the one used for the interconnect), choose 1 vote, click Ok
Click again on Add a Cluster Node, put the other node's hostname (interconnect), choose 1 vote, click Ok
Click on Failover Domains, Create a Failover Domain
Name the Domain as oracle_domain and click Ok
Click on Available Cluster Nodes and choose both nodes
Click on both check boxes
And adjust priority (priority is the preferable node for the cluster service) to db1=1, db2=2, then click Close
Click on Resources and then on Create Resource (4 times for file systems, 1 for VIP). Use the logical volumes created before. Usually we use also similar resource names for the mount points, as the name of the filesystem.
/orabin à ORACLE_HOME
/oradata à Datafiles+ControlFiles+Temporary
/redo1, /redo2 à RedoLogs (The setup of oracle redologs –multiplex as it is called- has to be done later by the Oracle DBA and can be done online. For our installation all redolog files will be saved in /oradata/….)
/arch à if database is to be configured in archive log mode, for online backups (not our case, Oracle DBA job also). If you know from the beginning of the project, that the database, will not be in archive log mode, then you do not need this file system at all)
Now Create the VIP IP resource and click Ok. The VIP should exist in /etc/hosts on both nodes. Check the monitor Link checkbox.
Click on Services and then on Create a Service
Choose TESTDB as service_name and click Ok
Make sure the Autostart This Service and Restart options are checked. Click on Failover Domain and choose oracle_domain
Click on Add a shared resource for this service (5 times, one for each resource). You must choose first the VIP and then the file systems
Now click on the VIP resource and then on the Attach a Shared Resource to the selection (4 times, one for each file system)
Click Close button
Go to FileàSave
Press Ok on the defaults
Close the system-config-cluster GUI
Enable automatic startup of cluster daemons (in both nodes db1 and db2)
Reboot both nodes and check (from the ILO console) that cluster is functional.
When both systems are up, perform the command clustat to verify the cluster proper configuration.
Create the quorum configuration (with direct editing of the cluster.conf), on none db1. Then propagate to db2 with ccs_tool
Create a quorum device with mkqdisk. Chose an empty device (normally quorum devices should be less than 1 GB of capacity).
Verify the quorum on node2 also
Modify the cluster.conf on db1 with the three white shadow lines (add them under the second line)
Modify the two_node=1 and change it to two_node=0 (two_node=1 can be used when we have no quorum)
Change the version in cluster.conf
Save the cluster.conf and propagate to db2
Configure the ILO cards on both nodes. Connect to ILO with the Web interface and make sure that you have setup a fence user/password. The user should have the right to connect and reset the server. ILO cards should be connected to a private switch, accessible by the sys admin for maintenance. The following printscreens, shows ILO config on one node. The same config should be done on the other ILO also.
Update cluster.conf on db1 with the ILO fence information. Insert the lines in white shadow. In our case, we used ilodb1 and ilodb2 for ILO hostnames, and we have updated /etc/hosts in both nodes with the ILO IPs.
Modify cluster.conf on db1 and add the lines in white shadow.
Change version
Save and propagate to db2
Test connectivity with ilodb1, ilodb2, db1, db2, VIP, GATEWAY from both nodes
[root@db1 ~]# ping ilodb1
PING ilodb1 (192.168.0.151) 56(84) bytes of data.
64 bytes from ilodb1 (192.168.0.151): icmp_seq=1 ttl=64 time=1.83 ms
64 bytes from ilodb1 (192.168.0.151): icmp_seq=2 ttl=64 time=0.580 ms
--- ilodb1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.580/1.207/1.834/0.627 ms
[root@db1 ~]# ping ilodb2
PING ilodb2 (192.168.0.152) 56(84) bytes of data.
64 bytes from ilodb2 (192.168.0.152): icmp_seq=1 ttl=64 time=1.50 ms
64 bytes from ilodb2 (192.168.0.152): icmp_seq=2 ttl=64 time=0.511 ms
--- ilodb2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.511/1.006/1.501/0.495 ms
[root@db1 ~]# ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.233 ms
--- 192.168.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.233/0.233/0.233/0.000 ms
[root@db1 ~]# ping db1
PING db1 (192.168.2.70) 56(84) bytes of data.
64 bytes from db1 (192.168.2.70): icmp_seq=1 ttl=64 time=0.063 ms
--- db1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.063/0.063/0.063/0.000 ms
[root@db1 ~]# ping db2
PING db2 (192.168.2.50) 56(84) bytes of data.
64 bytes from db2 (192.168.2.50): icmp_seq=1 ttl=64 time=0.178 ms
--- db2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.178/0.178/0.178/0.000 ms
[root@db1 ~]#
[root@db1 ~]# ping 192.168.0.90
PING 192.168.0.90 (192.168.0.90) 56(84) bytes of data.
64 bytes from 192.168.0.90: icmp_seq=1 ttl=64 time=0.065 ms
64 bytes from 192.168.0.90: icmp_seq=2 ttl=64 time=0.029 ms
--- 192.168.0.90 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.029/0.047/0.065/0.018 ms
Reboot both nodes, for the quorum device to start properly and to count in voting. qdiskd daemon should always start before the cman.
[root@db1 ~]# shutdown -ry now (reboot if it is going too slowly)
[root@db2 ~]# shutdown -ry now (reboot if it is going too slowly)
Note: Db1 and db2 are going to fence each other once, until they join to the cluster. Just
wait. In some cases you may have to update /etc/cluster/cluster.conf with
<fence_daemon post_fail_delay="0" post_join_delay="120"/>
Check cluster status, after both systems are up and running.
Oracle installation steps
Create user groups oinstall and dba, then create user oracle. Make sure they share the same userId and groupIDs between both nodes. This has to be done in both cluster nodes.
Modify .bash_profile in /home/oracle and add the following info (in both nodes). /home/oracle is assumed to be the home directory of user oracle.
export ORACLE_HOME=/orabin/oracle/product/11.2.0/TESTDB
export ORACLE_BASE=/orabin/oracle
export ORACLE_SID=TESTDB
export PATH=$ORACLE_HOME/bin:$PATH
Modify /etc/sysctl.conf in both nodes, and add (at the end of the file):
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
kernel.sem = 250 32000 100 128
Perform sysctl –p for the kernel to re-read the file online (both nodes)
Modify /etc/security/limits.conf and add (both nodes, end of the file):
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
NOTE:
The minimum swap space you need is 2GBs in both servers.
Change recursively the ownerships for all cluster file systems to belong to oracle.oinstall (both nodes)
[root@db2 /]# chown -R oracle.oinstall /orabin/ /oradata/ /redo1 /redo2
[root@db1 /]# chown -R oracle.oinstall /orabin/ /oradata/ /redo1 /redo2
Modify the /etc/profile and add the following (both nodes, end of file) lines (shown with white shadow)
Add the following line to the /etc/pam.d/login file, if it does not already exist.
session required pam_limits.so
Disable SELINUX on both nodes. Change the value to: disabled for the parameter SELINUX in the file /etc/selinux/config. This is for the chamge to be there after a system reboot. To change the SELINUX while the systems are online, do the following (both nodes)
echo 0 > /selinux/enforce
Move the oracle binary cds to /home/oracle and unzip them both (as user oracle) on node db1
Install the following rpms (or greater versions), for Oracle to install properly (on both nodes)
binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-2.5-24 (32 bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5 (32 bit)
glibc-headers-2.5
ksh-20060214
libaio-0.3.106
libaio-0.3.106 (32 bit)
libaio-devel-0.3.106
libaio-devel-0.3.106 (32 bit)
libgcc-4.1.2
libgcc-4.1.2 (32 bit)
libstdc++-4.1.2
libstdc++-4.1.2 (32 bit)
libstdc++-devel 4.1.2
make-3.81
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-2.2.11 (32 bit)
unixODBC-devel-2.2.11
unixODBC-devel-2.2.11 (32 bit)
You may check with the following command, to verify that the rpm is already installed (greater versions are ok). The following are some examples for a few rpms.
[root@db1 oracle]# rpm -qa|grep –i sysstat
sysstat-7.0.2-11.el5
[root@db1 oracle]# rpm -qa|grep -i libaio
libaio-devel-0.3.106-5
libaio-0.3.106-5 à THIS SHOWS THAT BOTH 32bit AND 64bit ARE INSTALLED BECAUSE IT IS DISPLAYED TWICE
libaio-0.3.106-5 à THIS SHOWS THAT BOTH 32bit AND 64bit ARE INSTALLED BECAUSE IT IS DISPLAYED TWICE
libaio-devel-0.3.106-5
[root@db1 oracle]# rpm -qa|grep -i libstdc++
compat-libstdc++-33-3.2.3-61
libstdc++-devel-4.1.2-51.el5
compat-libstdc++-33-3.2.3-61
libstdc++-4.1.2-51.el5
libstdc++-4.1.2-51.el5
If it is not installed then perform yum install rpm-name for all the above rpms.
NOTE: Another way to find and install the necessary rpms (if
for example Internet is not accessible), would be the CentOS DVD. You have to insert the DVD/CD into the CDROM, then go to /media/cdrom (check where the CD will auto mount itself), and then move to the directory Server. In this directory you should find the rpms that you should install in both systems.
Use the command: rpm –ivh rpm-name àto install
rpm –qa | grep –i rpm-name à to verify the existence
To install Oracle database in the cluster Virtual Hostname (oraprd.dyndns.org), we do the following trick.
- Login as root in db1 and issue: hostname oraprd.dyndns.org
- Login as oracle user, and start the installer, on node db1 (cd /home/oracle/database):
- ./runInstaller (Don't forget to have Xwin running on your laptop, and to export DISPLAY=yourIP:0.0)
- After the installation we reverse back the hostname to db (as user root)
- This trick is made only for oracle dbconsole to work properly in both nodes. Dbconsole binds itself with the actual hostname, and cannot be changed later.
Say yes to the warning regarding the email address
Click on Fix & Check again. Then go to /tmp/CVU_11.2.0.1.0_oracle and run (as root): ./runfixup.sh
Click Ok to continue
Click on finish. DO NOT FORGET TO COPY THE FIXES TO THE OTHER NODE
ALSO (db2). In our case, the problem was only two values in /etc/sysctl.conf We should copy these values to db2, in the file /etc/sysctl.conf and perfrom sysctl –p. The values are: kernel.sem = 250 100.Probably, if you use the values mentioned earlier (page 56), for the file sysctl.conf, you will have no problem.
If the oracle checks has no anymore findings, you can click on Finish
You may view the install log with the following command, all the time:
When prompted, run the following scripts as root, and click Ok
Make sure that all entries in /etc/sysctl.conf file from db1 are identical of those in db2. If necessary, copy the file from db1 to db2 and do: sysctl –p
Last lines from file /etc/sysctl.conf, in node db1 – Copy them to db2 also and do : sysctl -p
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
kernel.sem = 250 32000 100 128
Login as oracle and run netca (Network Configuration Assistant)
To create the Oracle Database, login as oracle and run dbca (Database Configuration Assistant)
Use the same passwords for all Oracle sys accounts: abc123
Click on Sample Schemas only if it is a test database
Accept the default memory distribution – This is normally good for most installations (If you need more memory for other resources, you may decrease the percentage to 30%
Oracle processes are the maximum number of connections allowed to connect to the Oracle server. This is DBA decision again, and depend on the project.
Click on Finish
Click on Ok
Click on Exit
Copy /etc/oratab and /etc/oraInst.loc files, from db1 to db2
Switch hostname back to normal (from oraprd to db1)
root@oraprd etc#hostname db1 (as user root)
Create a TESTDB.sh script on /etc/init.d on both nodes. This is the script used from the cluster to stop/start/status the TESTDB service. You can view this script from the /etc/init.d/TESTDB.sh
Open system-config-cluster and ad the TESTDB.sh script resource
Click on Edit Service Properties, choose the last resource (redo2) and click on Attached a Share Resource to the selection.
Select the TESTDB.sh script and click Ok
Click on Send to Cluster
Create the following soft links (as oracle user), in order for the oracle dbconsole to properly work. We use the real hostnames (db1, db2) and we link them to the Virtual host name (oraprd).
cd /orabin/oracle/product/11.2.0/TESTDB/oc4j/j2ee
ln -s OC4J_DBConsole_oraprd.dyndns.org_TESTDB/ OC4J_DBConsole_db1_TESTDB
ln -s OC4J_DBConsole_oraprd.dyndns.org_TESTDB/ OC4J_DBConsole_db2_TESTDB
cd /orabin/oracle/product/11.2.0/TESTDB
ln -s oraprd.dyndns.org_TESTDB/ db1_TESTDB
ln -s oraprd.dyndns.org_TESTDB/ db2_TESTD
As user oracle, go to $ORACLE_HOME/network/admin and insert the lines in White shadow, into the file listener.ora. Save the file and then do: lsnrctl reload
Also, in the same directory as before, vi the file tnsnames.ora and change the SERVICE_NAME to show SID
Perform two failover tests, as user root on the TESTDB service, to check Cluster functionality
clusvsadm –r TESTDB
clustat
cman_tool status
cman_tool nodes
Login into the database (via local firefox from db1 or db2), and check database health (DBA task). Passwords for sys/system accounts is abc123
END OF INSTALLATION
