In this post I'll
discuss full implementation of upgrading Oracle RAC 11.2.0.1 to 11.2.0.3, the
new RAC 11.2.0.3 will be installed on a new hardware (outplace upgrade).
This lengthy post (in
order to make it more beneficial) I divided it to FOUR major posts:
Part I OS
Installation, Filesystem preparation (OCFS2 on ISCSI)
->Covers Oracle Enterprise Linux 5.9 x86_64
installation, preparation of ISCSI storage, using OCFS2 to format the shared
filesystem.
->Covers Preparation of Linux OS for Oracle
,11.2.0.3 Grid Infrastructure & database software installation.
->Covers the creation of a standby database
being refreshed from the primary DB 11.2.0.1 taking advantage of a new feature
that a standby DB with an 11gr2 higher release can be refreshed from 11gr2
lower release.
Part IV Database
Upgrade from 11.2.0.1 to 11.2.03
->Covers switching over the new standby DB
resides on 11.2.0.3 server to act as a primary DB, Upgrade the new primary DB
from 11.2.0.1 to 11.2.0.3
Feel free to click on
the part you interested in.
Part I, II, III doesn't
require a downtime as they are being done on a different hardware as part of
our (outplace upgrade) only Part IV is the one will require a downtime window.
The whole implementation
may take less than two hours of downtime if every thing went smooth. but it
will take long hours of DBA work.
In this post I tried to
provide reference for each point. Also followed many recommendations
recommended by the Maximum Availability Architecture (MAA).
Part I OS
Installation, Filesystem preparation (OCFS2 on ISCSI)
The following are
"good to know" information before installing Oracle 11.2.0.3 on any
hardware:
Knowledge Requirements:
===================
RAC and Oracle
Clusterware Best Practices and Starter Kit (Linux) [Metalink Doc ID 811306.1]
RAC and Oracle
Clusterware Best Practices and Starter Kit (Platform Independent) [Metalink Doc
ID 810394.1]
Configuring raw devices
(singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5 [Metalink
Doc ID 465001.1]
Product Support
Lifetime:
===================
This document indicates
database release 11.2 premier support ends at Jan 2015 and the Extended support
ends at Jan 2018.
Patching Support
Lifetime:
===================
This document indicates
Oracle will continue provide security patches for 11.2.0.3 version till 27-Aug-2015.
Security patches means
(PSU, CPU, SPU patches). [Metalink Doc ID 742060.1]
Hardware Certification:
=================
RAC Technologies Matrix
for Linux Platforms:
The main link for
certification Matrix for other platforms: (Linux, Unix, Windows)
In this implementation
I'll install RAC on ISCSI NAS storage.
Now let's move from the
theoretical part to the technical steps...
Linux Requirements:
===============
If installing 11.2.0.3
on RHEL 5 x86_64:
The minimum requirement
is Red Hat Enterprise Linux 5 Update 5 with the Unbreakable Enterprise Kernel
2.6.32 or later.
Partitioning requirement
on the server’s local hard disk: [Minimum!]
=======================================
Local hard disk will
contain Linux, Grid Infrastructure and database installed binaries.
/u01 => 10G
Free space to hold the installation files (GI+DB). I recommend at least 30G to
hold future generated logs.
/tmp => 1G
Free space.
SWAP =>
RAM=32G which is >8G, SWAP= 75% of RAM = 24G
/dev/shm => must
be greater than the sum of MEMORY_MAX_TARGET for all instance if you will use
the new 11g feature Automatic Memory Management by setting parameters
memory_max_target & memory_target to a specific value which will handle
the memory size of SGA & PGA together.
More about /dev/shm:
-According to Oracle
Support /dev/shm will not be able to be greater than 50% of the RAM installed
on the server.
-Make /dev/shm size =
Memory Size installed on the server. or at least the sum of all DBs
Memory_max_target on the server.
-/dev/shm must be exist
if you will use 11g new feature Auto Memory Management by setting
memory_max_target parameter.
-If /dev/shm isn't exist
or isn't properly sized the database will pop up this error when starting up if
memory_max_target parameter has been set:
ORA-00845:
MEMORY_TARGET not supported on this system.
-Oracle will create
files under /dev/shm upon instance startup and will be removed automatically
after instance shutdown.
-Oracle will use these
files to manage the memory size dynamically between SGA and PGA.
-It's recommended to
have the /dev/shm configured with the "tmpfs" option instead of
"ramfs", as ramfs is not supported for Automatic Memory Management
AMM:
# df -h /dev/shm
Filesystem
Size Used Avail Use% Mounted on
tmpfs
16G 0 16G 0% /dev/shm
BTW I'm not using this
feature I'm still stick with sga_taget & pga_aggregate_target. :-)
------------------------------
Linux OS Installation: Both Nodes (Estimated time: 3 hours)
------------------------------
Note: Install a fresh
Linux installation on all RAC nodes, DON'T clone the installation from node to
other in purpose of saving the time.
FS Layout:
>>>>>>>>>
The whole disk space is
300G
Filesystem
Size(G) Size(M) used in setup
----------
---- ---------------------
/boot
1G 1072 --Force to Be Primary Partition.
Swap
24G 24576 --Force to Be Primary Partition, 75% of RAM.
/dev/shm
30G 30720 --Force
to Be Primary Partition.
/
20G 20480
/u01
70G 73728
/home
10G 10547
/tmp
5G 5240
/var
10G 10547
/u02
95G The rest of space
Note:
If you will
install ISCSI drive avoid making a separate partition for /usr , ISCSI drive
will prevent system from booting.
Packages selection
during Linux installation:
>>>>>>>>>>>>>
Desktop Environment:
# Gnome Desktop Environment
Applications:
#Editors -> VIM
Development:
# Development Libraries.
# Development Tools
# GNOME software development
# Java Development
# Kernel Development
# Legacy Software Development
# X Software Development
Servers:
# Legacy Network Server -> Check only:
rsh-server,xinetd
# PostgreSQL -> Check only:
UNIXODBC-nnn
# Server Configuration Tools -> Check
All
Base System:
# Administration Tools.
# Base -> Un check bluetooth,wireless
packs Check-> Device mapper multipath
# Java
# Legacy Software Support
# System Tools -> Check also: OCFS2
packages
# X Window System
FIREWALL & SELINUX
MUST BE STOPPED. [Note ID 554781.1]
I've uploaded OEL 5.9
installation snapshots in this link:
populate /etc/hosts with
the IPs and resolved names:
=======================================
# vi /etc/hosts
#You must keep
127.0.0.1 localhost, if removed VIP will not work !!!
#cluster_scan,Public
and VIP should be in the same subnet.
127.0.0.1 localhost localhost.localdomain
#Public:
172.18.20.1 ora1123-node1
node1 n1
172.18.20.2 ora1123-node2
node2 n2
#Virtual:
172.18.20.3 ora1123-node1-vip
node1-vip n1-vip
172.18.20.4 ora1123-node2-vip
node2-vip n2-vip
#Private:
192.168.10.1
ora1123-node1-priv n1-priv node1-priv
192.168.10.2
ora1123-node2-priv n2-priv node2-priv
#Cluster:
172.18.20.10
cluster-scan
#NAS
172.20.30.100
nas nas-server
#11.2.0.1 Servers:
10.60.60.1 ora1121-node1
old1 #the current 11.2.0.1 Node1
10.60.60.2 ora1121-node2
old2 #the current 11.2.0.1 Node2
I've added RAC node
names, VIP and private IPs and it's resolved names for both nodes and guess
what I'm resolving also the cluster scan in /etc/hosts, keep it a secret don't
till Larry :-)
Actually I'm still not
convinced with using the SCAN feature, if you will use it in your setup just
ask the network admin to resolve at least three SCAN IPs in the DNS to the
cluster scan name you will use.
This document will help
you understanding the SINGLE CLIENT ACCESS NAME (SCAN):
Upgrade the KERNEL:
==================
-Subscribe The new
servers in ULN network.
-Upgrade the Kernel to
the latest version.
Ensure that
/etc/resolv.conf is equipped with the DNS entry and you are connected to the
internet, once this task is done if you don't have a need to connect the
servers to the internet make sure the servers are not connecting anymore to the
internet for security reason.
On linux server:
-------------------
# up2date --register
Install key? Yes
put this information:
login: xxxxxx
pass: xxxxxx
CSI#: xxxxxx
In case you still cannot
establish a connection with ULN You can use the IP 141.146.44.24 instead of
address linux-update.oracle.com in "Network Configuration" button.
Also: in
/etc/sysconfig/rhn/up2date :
You
can change this line:
noSSLServerURL=http://linux-update.oracle.com/XMLRPC to
noSSLServerURL=http://141.146.44.24/XMLRPC
and
this line:
serverURL=https://linux-update.oracle.com/XMLRPC to
serverURL=https://141.146.44.24/XMLRPC
Then proceed with
updating the kernel from the same GUI or from command line as shown below:
up2date -d @ --> To download the updated packages
up2date @ --> To install the updated packages
I'm putting the symbol @
for skipping the GUI mode and continue with CLI.
Configure YUM with ULN:
--------------------------------
# cd /etc/yum.repos.d
# wget
http://public-yum.oracle.com/public-yum-el5.repo
# vi
public-yum-el5.repo
Modify the following:
Under both paragraphs:
[el5_latest] & [ol5_UEK_latest] modify enabled=0 to enabled=1
An excerpt:
[el5_latest]
name=Oracle Linux
$releasever Latest ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1
[ol5_UEK_latest]
name=Latest
Unbreakable Enterprise Kernel for Oracle Linux $releasever ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/UEK/latest/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1
Network configuration:
================
Node1:
--------
# cat
/etc/sysconfig/network-scripts/ifcfg-eth0 #=>Public
DEVICE=eth0
BOOTPROTO=static
BROADCAST=172.18.20.255
IPADDR=172.18.20.1
NETMASK=255.255.255.0
NETWORK=172.18.20.0
ONBOOT=yes
# cat /etc/sysconfig/network-scripts/ifcfg-eth1 #=>ISCSI NAS
DEVICE=eth1
BOOTPROTO=static
BROADCAST=172.20.30.255
IPADDR=172.20.30.101
NETMASK=255.255.255.0
NETWORK=172.20.30.0
ONBOOT=yes
# cat /etc/sysconfig/network-scripts/ifcfg-eth3 #=>Private
DEVICE=eth3
BOOTPROTO=static
BROADCAST=192.168.10.255
IPADDR=192.168.10.1
NETMASK=255.255.255.0
NETWORK=192.168.10.0
ONBOOT=yes
Node2:
--------
# cat
/etc/sysconfig/network-scripts/ifcfg-eth0 #=>Public
DEVICE=eth0
BOOTPROTO=static
BROADCAST=172.18.20.255
IPADDR=172.18.20.2
NETMASK=255.255.255.0
NETWORK=172.18.20.0
ONBOOT=yes
# cat /etc/sysconfig/network-scripts/ifcfg-eth1 #=>ISCSI NAS STORAGE
DEVICE=eth1
BOOTPROTO=static
BROADCAST=172.20.30.255
IPADDR=172.20.30.102
NETMASK=255.255.255.0
NETWORK=172.20.30.0
ONBOOT=yes
# cat /etc/sysconfig/network-scripts/ifcfg-eth3 #=>Private
DEVICE=eth3
BOOTPROTO=static
BROADCAST=192.168.10.255
IPADDR=192.168.10.2
NETMASK=255.255.255.0
NETWORK=192.168.10.0
ONBOOT=yes
-----------------------------
Filesystem Preparation:
----------------------
RAC servers will connect
to the NAS shared storage using ISCSI protocol.
ISCSI Configuration:
Required packages:
# rpm -q
iscsi-initiator-utils
# yum install
iscsi-initiator-utils
To get ISCSI aware that
LUNs are being accessed simultaneously by more than one node at the same time
and to avoid LUNs corruption, use one of the following ways (A or B):
A) Generate IQN number
or
B) Setup username and
password of ISCSI storage
I'll explain both of
them:
A) Generate IQN number (ISCSI Qualified Name) in linux for each node to
be saved inside NAS configuration console:
On Node1:
Generate an IQN number:
# /sbin/iscsi-iname
iqn.1988-12.com.oracle:9e963384353a
Note: the last portion
of IQN after semicolon ":" is editable and can be changed to the node
name, I mean instead of "9e963384353a" you can rename it
"node1", no space
allowed in the name.
Now insert the generated
IQN to /etc/iscsi/initiatorname.iscsi
# vi
/etc/iscsi/initiatorname.iscsi
#Note that last portion
of the IQN is modifyiable (modify it to meaningful name)
InitiatorName=iqn.1988-12.com.oracle:node1
Do the same on Node2:
On Node2:
# /sbin/iscsi-iname
iqn.1988-12.com.oracle:18e6f43d73ad
# vi
/etc/iscsi/initiatorname.iscsi
#Note that last portion
of the IQN is modifyiable (modify it to meaningful name)
InitiatorName=iqn.1988-12.com.oracle:node2
Put the same IQN you
already inserted in /etc/iscsi/initiatorname.iscsi on both nodes in the NAS
administration console for each LUN will be accessed by both nodes.
(This should be done by
the Storage Admin)
B) Set up a username and
password for ISCSI storage:
# vi
/etc/iscsi/iscsid.conf
node.session.auth.username
=
node.session.auth.password
=
discovery.sendtargets.auth.username
=
discovery.sendtargets.auth.password
=
Start the iscsi service:
# /etc/init.d/iscsi
start
Same Username &
password should be configured in the NAS administration console.(This should be
done by the Storage Admin)
Continue configuring the
ISCSI:
=======================
Turn on the iscsi
service to start after booting the machine:
# chkconfig iscsi on
Discover the target
LUNs:
# service iscsi start
# iscsiadm -m
discovery -t sendtargets -p 172.20.30.100
# (cd
/dev/disk/by-path; ls -l *iscsi* | awk '{FS=" "; print $9 "
" $10 " " $11}')
Whenever iscsid
discovers new target, it will add corresponding information in the following
directory:
# ls -lR
/var/lib/iscsi/nodes/
# service iscsi
restart
Create Persistent
Naming: (NON Multipath Configuration)
===================
Every time the machine
or ISCSI service restart the partitions source names /dev/sd* will change e.g.
/data1 will point to /dev/sdc instead of /dev/sda, the thing we cannot live
with it at all.
Note: I only have one
physical path "NIC" connecting to the NAS storage, so I apply non
multipath configuration.
1) Whitelist all SCSI
devices:
--
-----------------------------
# vi
/etc/scsi_id.config
#Add the following
lines:
vendor="ATA",options=-p
0x80
options=-g
2) Get the names of LUNs
and it's device name:
--
--------------------------------------------------
# (cd
/dev/disk/by-path; ls -l *iscsi* | awk '{FS=" "; print $9 "
" $10 " " $11}')
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-archive1-lun-0
-> ../../sdn
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-archive2-lun-0
-> ../../sdr
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-backupdisk-lun-0
-> ../../sdi
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-control1-lun-0
-> ../../sdl
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-control2-lun-0
-> ../../sda
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-data1-lun-0
-> ../../sdp
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-index1-lun-0
-> ../../sdo
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr1-lun-0
-> ../../sdq
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr2-lun-0
-> ../../sde
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr3-lun-0
-> ../../sdf
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-redo1-lun-0
-> ../../sdb
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-redo2-lun-0
-> ../../sdm
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-temp1-lun-0
-> ../../sdh
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-undo1-lun-0
-> ../../sdj
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-undo2-lun-0
-> ../../sdc
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting1-lun-0
-> ../../sdk
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting2-lun-0
-> ../../sdg
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting3-lun-0
-> ../../sdd
3) Get the drives UUID:
-- ------------------------
scsi_id -g -s
/block/sdn
scsi_id -g -s
/block/sdr
scsi_id -g -s
/block/sdi
scsi_id -g -s
/block/sdl
scsi_id -g -s
/block/sda
scsi_id -g -s
/block/sdp
scsi_id -g -s
/block/sdo
scsi_id -g -s
/block/sdq
scsi_id -g -s
/block/sde
scsi_id -g -s
/block/sdf
scsi_id -g -s
/block/sdb
scsi_id -g -s
/block/sdm
scsi_id -g -s
/block/sdh
scsi_id -g -s
/block/sdj
scsi_id -g -s
/block/sdc
scsi_id -g -s
/block/sdk
scsi_id -g -s
/block/sdg
scsi_id -g -s
/block/sdd
These UUIDs are the
consistent identifiers for the devices, we will use them in the next step.
4) Create the file
/etc/udev/rules.d/04-oracle-naming.rules with the following format:
--
--------------------------------------------------------------------------------------
# vi
/etc/udev/rules.d/04-oracle-naming.rules
#Add a line for each
device specifying the device name & it's UUID:
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d003000000000",
NAME="archive1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d004000000000",
NAME="archive2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d002000000000",
NAME="backupdisk"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d001000000000",
NAME="control1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d005000000000",
NAME="control2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d006000000000", NAME="data1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d007000000000", NAME="index1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d008000000000", NAME="ocr1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d009000000000", NAME="ocr2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d010000000000", NAME="ocr3"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d011000000000", NAME="redo1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d012000000000", NAME="redo2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d013000000000", NAME="temp1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d014000000000", NAME="undo1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d015000000000", NAME="undo2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d016000000000", NAME="voting1"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d017000000000", NAME="voting2"
KERNEL=="sd*",
BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -s /block/%k",
RESULT=="360014052e3032700063d018000000000", NAME="voting3"
# service iscsi
restart
5) Check the
configuration:
--
-----------------------------
Now new files name under
/dev should be e.g. /dev/archive1 instead of /dev/sdn
Note: fdisk -l will not
show the new NAS devices anymore, don't worry use the following:
#(cd
/dev/disk/by-path; ls -l *iscsi* | awk '{FS=" "; print $9 "
" $10 " " $11}')
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-archive1-lun-0
-> ../../archive1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-archive2-lun-0
-> ../../archive2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-backupdisk-lun-0
-> ../../backupdisk
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-control1-lun-0
-> ../../control1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-control2-lun-0
-> ../../control2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-data1-lun-0
-> ../../data1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-index1-lun-0
-> ../../index1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr1-lun-0
-> ../../ocr1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr2-lun-0
-> ../../ocr2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-ocr3-lun-0
-> ../../ocr3
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-redo1-lun-0
-> ../../redo1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-redo2-lun-0
-> ../../redo2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-temp1-lun-0
-> ../../temp1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-undo1-lun-0
-> ../../undo1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-undo2-lun-0
-> ../../undo2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting1-lun-0
-> ../../voting1
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting2-lun-0
-> ../../voting2
ip-172.20.30.100:3260-iscsi-iqn.2013-7.VLA-NAS03:pefms-voting3-lun-0
-> ../../voting3
also test UDEV rule:
# udevtest /block/sdb
| grep udev_rules_get_name
udev_rules_get_name:
rule applied, 'sdb' becomes 'ocr2'
...
OCFS2 Configuration:
Required Packages:
OCFS2 packages should be
installed during Linux installation, if you selected the right packages.
If you didn't do so, you
can download and install the required OCFS2 packages using the following
commands:
# up2date --install
ocfs2-tools ocfs2console
# up2date --install
ocfs2-`uname -r`
1) populate
/etc/ocfs2/cluster.conf settings:
-
-------------------------------------
In the OCFS2 configuration
I'll use the heartbeat NICs (private) not the public ones.
# mkdir -p /etc/ocfs2/
# vi
/etc/ocfs2/cluster.conf
node:
ip_port = 7000
ip_address = 192.168.10.1
number = 0
name = ora1123-node1
cluster = ocfs2
node:
ip_port = 7000
ip_address = 192.168.10.2
number = 1
name = ora1123-node2
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
Options:
ip_port:
The Default Can be changed to unused port.
ip_address: Using the
private interconnect is highly recommended as it's supposed to be a private
network between cluster nodes only.
number:
Node unique number from 0-254
name:
The node name needs to match the hostname without the domain name.
cluster:
Name of the cluster.
node_count: Number of
the nodes in the cluster.
BEWARE: During editing
the file be-careful, parameters must start after a tab, a blank space must
separate each value.
2) Timeout
Configuration:
-
-----------------------------
The O2CB cluster stack
uses these timings to determine whether a node is dead or alive. Keeping
default values is recommended.
# /etc/init.d/o2cb
configure
Load O2CB driver on
boot (y/n) [n]: y
Cluster stack backing
O2CB [o2cb]:
Cluster to start on
boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead
threshold (>=7) [31]: 61
Specify network idle
timeout in ms (>=5000) [30000]: 60000
Specify network
keepalive delay in ms (>=1000) [2000]:
Specify network
reconnect delay in ms (>=2000) [2000]:
Heartbeat Dead
Threshold: is the number of two-second iterations before a node is considered
dead.61 is recommended for multipath users, for my setup I'll
set the timeout to
120sec.
Network Idle Timeout:
The time in milliseconds before a network connection is considered
dead.recommended 60000ms
configured the cluster
to load on boot:
-------------------------------------------
# chkconfig --add o2cb
# chkconfig --add
ocfs2
# /etc/init.d/o2cb
load
# /etc/init.d/o2cb
start ocfs2
Filesystem Partitioning:
OCFS2
==================
As per the labels on the
NAS disk names, I'll assign same names with OCFS2.
# fdisk -l |grep /dev
# (cd
/dev/disk/by-path; ls -l *iscsi* | awk '{FS=" "; print $9 "
" $10 " " $11}')
Formating:
--------------
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L ocr1 /dev/ocr1
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L ocr2 /dev/ocr2
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L ocr3 /dev/ocr3
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L voting1 /dev/voting1
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L voting2 /dev/voting2
# mkfs.ocfs2 -F -b 4K
-C 32K -N 2 -L voting3 /dev/voting3
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L redo1 -J size=64M
/dev/redo1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L redo2 -J size=64M
/dev/redo2
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L control1 -J size=64M
/dev/control1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L control2 -J size=64M
/dev/control2
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L archive1 -J size=64M
/dev/archive1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L archive2 -J size=64M /dev/archive2
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L undo1 -J size=64M
/dev/undo1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L undo2 -J size=64M
/dev/undo2
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L data1 -J size=64M
/dev/data1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L index1 -J size=64M /dev/index1
# mkfs.ocfs2 -F -b 4k
-C 8k -N 2 -L temp1 -J size=64M /dev/temp1
# mkfs.ocfs2 -F -b 4k
-C 1M -N 2 -L backupdisk -J size=64M
/dev/backupdisk
Options:
-F If the device was
previously formatted by OCFS to overwrite the data.
-b blocksize from 512 to
4k(default), 4k recommended (small block size mean smaller maxsize,
maxsize=2^32*blocksize means with blocksize=4096 maxsize=16T).
-C clustersize from
4k(default) to 1M, 4k recommended EXCEPT for DBFILES partition it should =
Database Block Size =8k
For backup
storage filesystem holding RMAN, dump files, use bigger clustersize.
128k
recommended as a default clustersize if you're not sure what clustersize to
use.
-N #Node slots, number
of nodes can mount the volume concurrently, it's recommended to set it bigger
than required,e.g. if you have two nodes set it to 4, this
parameter can be
increased later using tunefs.ocfs2 but this practice can lead to bad
performance.
-L lablel name, labeling
the volume allow consistent "presistent" naming across the cluster.
even if you're using ISCSI.
-J Journal size, 256
MB(default), recommeded (64MB for datafiles, 128MB for vmstore and 256MB for
mail).
-T filesystem-type
(datafiles,mail,vmstore)
(datafiles) recommended for database FS will set (blocksize=4k,
clustersize=128k, journal size=32M)
(vmstore)
recommended for backup FS will set (blocksize=4k, clustersize=128k,
journal size=128M) .
Note: For the
filesystems that hold the database files set the cluster size -C 8k
For
the filesystems that hold backup files set the cluster size -C 1M
If
you're not sure what cluster size to use, use 128k, it proven reasonable
trade-off between wasted space and performance.
Mounting the partitions:
-----------------------------
mkdir /ora_redo1
mkdir /ora_backupdisk
mkdir /ora_undo1
mkdir /ora_undo2
mkdir /ora_control2
mkdir /ora_control1
mkdir /ora_archive1
mkdir /ora_redo2
mkdir /ora_temp1
mkdir /ora_index1
mkdir /ora_archive2
mkdir /ora_data1
mkdir /ora_ocr1
mkdir /ora_ocr2
mkdir /ora_ocr3
mkdir /ora_voting1
mkdir /ora_voting2
mkdir /ora_voting3
chown -R oracle:oinstall /ora*
chmod 750 /ora*
Mounting the partitions
automatically when system restart:
-----------------------------------------------------------------
vi /etc/fstab
LABEL=ocr1 /ora_ocr1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=ocr2 /ora_ocr2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=ocr3 /ora_ocr3 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=voting1 /ora_voting1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=voting2 /ora_voting2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=voting3 /ora_voting3 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=control1 /ora_control1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=control2 /ora_control2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=redo1 /ora_redo1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=redo2 /ora_redo2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=archive1 /ora_archive1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=archive2 /ora_archive2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=temp1 /ora_temp1 ocfs2 _netdev,datavolume,nointr
0 0
LABEL=undo1 /ora_undo1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=undo2 /ora_undo2 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=index1 /ora_index1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=data1 /ora_data1 ocfs2
_netdev,datavolume,nointr 0 0
LABEL=backupdisk
/ora_backupdisk ocfs2 _netdev 0 0
Partitions mount
options:
>>>>>>>>>>>>>>>>>
_netdev: mandatory,
prevent attempting to mount the filesystem until the network has been enabled
on the system.
datavolume: force using
direct I/O, used with FS contain Oracle data files, control files,
redo/archive, voting/OCR disk. same behavior when using init.ora
filesystemio_options.
datavolume mount option MUST NOT be used on volumes
hosting the Oracle home or Oracle E-Business Suite or any other use.
nointr: default, blocks
signals from interrupting certain cluster operations, disable interrupts.
rw: default, mount the
FS in read write mode.
ro: mount the FS in read
only mode.
noatime: default,
disable access time updates, improve the performance (important for DB/cluster
files).
atime_quantum=: update
atime of files every 60 second(default), degrades the performance.
commit=: optional, sync
all data every 5 seconds(default), degrades the performance. in case of failure
you will lose last 5 seconds of work (Filesystem will
not be damaged, thanks
to journaling). higher assigned value improves the performance with more data
loss risk.
After adding the values
inside /etc/hosts you can mount the partitions using these commands:
# mount -a
OR:
# mount -L
"temp1" /ora_temp1
# df -h
Filesystem
Size Used Avail Use% Mounted on
/dev/cciss/c1d0p6
20G 3.9G 15G 22% /
/dev/cciss/c1d0p10
98G 7.0G 86G 8% /u02
/dev/cciss/c1d0p9
5.0G 139M 4.6G 3% /tmp
/dev/cciss/c1d0p8
10G 162M 9.4G 2% /home
/dev/cciss/c1d0p7
10G 629M 8.9G 7% /var
/dev/cciss/c1d0p2
16G 0 16G 0% /dev/shm
/dev/cciss/c1d0p5
70G 180M 66G 1% /u01
/dev/cciss/c1d0p1
1003M 76M 876M 8% /boot
tmpfs
16G 0 16G
0% /dev/shm
/dev/sdm
1.0G 143M 882M 14%
/ora_ocr1
/dev/sdd
1.0G 143M 882M 14%
/ora_ocr2
/dev/sde
1.0G 143M 882M 14%
/ora_ocr3
/dev/sdk
1.0G 143M 882M 14%
/ora_voting1
/dev/sdi
1.0G 143M 882M 14%
/ora_voting2
/dev/sdg
1.0G 143M 882M 14%
/ora_voting3
/dev/sdl
10G 151M 9.9G 2% /ora_control1
/dev/sda
10G 151M 9.9G 2%
/ora_control2
/dev/sdb
10G 151M 9.9G 2%
/ora_redo1
/dev/sdr
10G 151M 9.9G 2%
/ora_redo2
/dev/sdp
300G 456M 300G 1%
/ora_archive1
/dev/sdn
300G 456M 300G 1%
/ora_archive2
/dev/sdf
60G 205M 60G 1%
/ora_temp1
/dev/sdj
40G 184M 40G 1%
/ora_undo1
/dev/sdc
40G 184M 40G 1%
/ora_undo2
/dev/sdo
200G 349M 200G 1% /ora_index1
/dev/sdq
400G 563M 400G 1%
/ora_data1
/dev/sdh
500G 674M 500G 1%
/ora_backupdisk
Performance Tip: Ensure
updatedb is not running on OCFS2 partitions, by adding "OCFS2"
keyword to "PRUNEFS =" list in file /etc/updatedb.conf
///////////////////////////////////////////////////////////////
In case of using ASM for the shared storage
(very quick guide)
///////////////////////////////////////////////////////////////
Note: Don't use persistent naming unless you
finish configuring ASM first.
Install ASMLib 2.0 Packages:
---------------------------
# rpm -qa --queryformat
"%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep oracleasm | sort
oracleasm-2.6.18-348.el5-2.0.5-1.el5 (x86_64)
oracleasmlib-2.0.4-1.el5 (x86_64)
oracleasm-support-2.1.7-1.el5 (x86_64)
Configure ASMLib:
----------------
# /usr/sbin/oracleasm configure -i
Default user to own the driver interface []:
oracle
Default group to own the driver interface []:
dba
Start Oracle ASM library driver on boot (y/n)
[n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
# /usr/sbin/oracleasm init
Use FDISK to create RAW partition for each
disk:
-----------------------------------------------
# fdisk /dev/sdn
n
p
1
w
Do the same for other disks....
Commit your changes without the need to
restart the system using this command:
# partprobe
Create ASM Disks:
----------------
# /usr/sbin/oracleasm createdisk OCR1
/dev/sdn1
# /usr/sbin/oracleasm createdisk OCR2
/dev/sdd1
# /usr/sbin/oracleasm createdisk OCR3
/dev/sde1
# /usr/sbin/oracleasm createdisk voting1
/dev/sdk1
# /usr/sbin/oracleasm createdisk voting2
/dev/sdi1
# /usr/sbin/oracleasm createdisk voting3
/dev/sdj1
... and so on
SCAN ASM Disks:
--------------
# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCR1"
Instantiating disk "OCR2"
....
# /usr/sbin/oracleasm listdisks
OCR1
...
# oracleasm querydisk /dev/sdn1
Diskgroup creation will be done from the
installer.
////////////////////////////////////////////////////////////////////
In case you want to
use RAW DEVICES for the shared storage:
Note that starting
with 11gr2 using DBCA or the installer to store Oracle Clusterware or Oracle
Database files on block or raw devices is not supported.
Next:
In Part II I’ll
continue with OS preparation for RAC setup, Grid Infrastructure and Database
installation on the new servers.
No comments:
Post a Comment