Showing posts with label Data Guard. Show all posts
Showing posts with label Data Guard. Show all posts

Tuesday, February 21, 2023

Copy Archivelogs From Standby DR to Primary

You may think the title of this article is wrong; what is the scenario that requires copying archivelogs from Standby to Primary?

I ran into a situation on a primary RAC DB data guarded by a standby RAC DB, where dozens of archivelogs were needed for goldengate replication got deleted by Oracle because of space pressure on Flash Recovery Area FRA. For my good luck, those archivelogs were still available on the Standby DB and hence in this article I'll demonstrate how to restore them back on the primary DB.

Note: Both Primary and Standby DBs are two nodes RAC, this explains why I've to specify the thread in most of RMAN commands.

Backup the Archivelogs on the Standby:

I managed to figure out the archivelogs sequence numbers I need to copy from the Standby DB and hence I used the following RMAN command to back up the needed Archivelogs on the Standby DB:

RUN{
ALLOCATE CHANNEL ch1 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch2 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch3 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch4 DEVICE TYPE DISK ;
backup as compressed backupset
archivelog sequence between 112700 and 112902 thread 1
format '/backup/archives/%d_%s_%p_%c_%t.arc.bkp'
archivelog sequence between 102590 and 102669 thread 2 format '/backup/archives/%d_%s_%p_%c_%t.arc.bkp';
}

Note: Using "as compressed backupset" option will shrink the backup file size, and it doesn't require a license.

Then, Copy the backed up Archivelogs from the Standby to one of the Primary DB nodes:
# scp /backup/archives/* oracle@primary_n1:/backup/archives


Restore the Archivelogs on the Primary:

On the Primary DB node where the backupfiles are copied, use the following command to Catalog the backup files, so they will be aware of by the Restore command:

RMAN> catalog start with '/backup/archives/';

On the same Primary Node I used the following RMAN command to restore the archives logs to their default location which is +RECO ASM diskgroup in my case:

RUN{
ALLOCATE CHANNEL ch1 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch2 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch3 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch4 DEVICE TYPE DISK ;
restore
archivelog from logseq=112700 until logseq=112902 thread 1
archivelog from logseq=102590 until logseq=102669 thread 2;
}

In case you want to restore the archivelogs to a non-default location i.e. /backup/archives/restored you can use "set archivelog destination" command as follows:

RUN{
ALLOCATE CHANNEL ch1 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch2 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch3 DEVICE TYPE DISK ;
ALLOCATE CHANNEL ch4 DEVICE TYPE DISK ;
set archivelog destination to '/backup/archives/restored';
restore
archivelog from logseq=112700 until logseq=112902 thread 1
archivelog from logseq=102590 until logseq=102669 thread 2;
}

 

Tuesday, October 11, 2022

ORA-19909: datafile 1 belongs to an orphan incarnation, After Recovering the Standby DB from Service

Scenario: 

On a 19.5 DB, I've opened the Standby in Read Write mode (ACTIVATED not SNAPSHOT mode) for testing purpose for 24 hours, which make it lag too way behind the primary and needs shipping of 700 GB of Archivelogs which will take ages to complete, So I followed the new feature of recovering the standby DB from RMAN incremental backup using the following one command:

RMAN> RECOVER STANDBY DATABASE FROM SERVICE PRC;

Problem:

After the completion of the recovery, --which took a couple of hours, I got this error in the DB alert log:

2022-10-05T18:05:52.215176+00:00
Errors in file /u01/oracle/diag/rdbms/prs/prs/trace/prs_mz00_15686.trc:
ORA-01110: data file 93: '+DATA/PRS/DATAFILE/p_data.278.1068357141'
2022-10-05T18:05:52.321877+00:00

When trying to start the recovery, it crashed, and I got the following error in the alertlog:

SQL> recover managed standby database using current logfile nodelay disconnect;

Excerpt from the alertlog:

2022-10-05T17:57:23.619258+00:00
PR00 (PID:11782): Managed Standby Recovery starting Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 6629958934149) is orphaned on incarnation#=2
PR00 (PID:11782): MRP0: Detected orphaned datafiles!
2022-10-05T17:57:23.804939+00:00
Errors in file /u01/oracle/diag/rdbms/prs/prs/trace/prs_pr00_11782.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '+DATA/PRS/DATAFILE/system.378.1040043713'
PR00 (PID:11782): Managed Standby Recovery not using Real Time Apply
stopping change tracking


Analysis:

As the error mentioning the incarnation, I went to RMAN to check all incarnations of the DB:

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PRD  1352250453       PARENT  1          17-APR-19
2       2       PRD  1352250453       PARENT  1920977    07-MAY-20

3       3       PRD  1352250453       ORPHAN  6472651405014 12-MAY-20
4       4       PRD  1352250453       ORPHAN  6473945178918 15-JUN-20
5       5       PRD  1352250453       ORPHAN  6475796134931 07-JUL-20
6       6       PRD  1352250453       ORPHAN  6476092131516 10-JUL-20
7       7       PRD  1352250453       ORPHAN  6476262305573 12-JUL-20
8       8       PRD  1352250453       ORPHAN  6476996897342 22-JUL-20
9       9       PRD  1352250453       ORPHAN  6476997075925 22-JUL-20
10      10      PRD  1352250453       ORPHAN  6476998329789 22-JUL-20
11      11      PRD  1352250453       ORPHAN  6477049189900 22-JUL-20
12      12      PRD  1352250453       ORPHAN  6477792328248 03-AUG-20
13      13      PRD  1352250453       ORPHAN  6477896848188 04-AUG-20
14      14      PRD  1352250453       ORPHAN  6477898103437 04-AUG-20
15      15      PRD  1352250453       ORPHAN  6477898886110 04-AUG-20
16      16      PRD  1352250453       ORPHAN  6477907643794 04-AUG-20
17      17      PRD  1352250453       ORPHAN  6477908164587 04-AUG-20
18      18      PRD  1352250453       ORPHAN  6477908493572 04-AUG-20
19      19      PRD  1352250453       ORPHAN  6477908951719 04-AUG-20
20      20      PRD  1352250453       ORPHAN  6478017068793 05-AUG-20
21      21      PRD  1352250453       ORPHAN  6478101320713 06-AUG-20
22      22      PRD  1352250453       ORPHAN  6478223633690 07-AUG-20
23      23      PRD  1352250453       ORPHAN  6478226677036 07-AUG-20
24      24      PRD  1352250453       ORPHAN  6478261288240 07-AUG-20
25      25      PRD  1352250453       ORPHAN  6478262694239 07-AUG-20
26      26      PRD  1352250453       ORPHAN  6478358471117 08-AUG-20
27      27      PRD  1352250453       ORPHAN  6478434623641 09-AUG-20
28      28      PRD  1352250453       ORPHAN  6478438625990 09-AUG-20
29      29      PRD  1352250453       ORPHAN  6478446183062 09-AUG-20
30      30      PRD  1352250453       ORPHAN  6480950862854 31-AUG-20
31      31      PRD  1352250453       ORPHAN  6482153817265 09-SEP-20
32      32      PRD  1352250453       ORPHAN  6482271564751 10-SEP-20
33      33      PRD  1352250453       ORPHAN  6482286544445 10-SEP-20
34      34      PRD  1352250453       ORPHAN  6482723618944 14-SEP-20
35      35      PRD  1352250453       ORPHAN  6482800530717 15-SEP-20
36      36      PRD  1352250453       ORPHAN  6482918772612 16-SEP-20
37      37      PRD  1352250453       ORPHAN  6483074592719 17-SEP-20
38      38      PRD  1352250453       ORPHAN  6483533227915 21-SEP-20
39      39      PRD  1352250453       ORPHAN  6485299063993 05-OCT-20
40      40      PRD  1352250453       ORPHAN  6486322720680 13-OCT-20
41      41      PRD  1352250453       ORPHAN  6489734338575 10-NOV-20
42      42      PRD  1352250453       ORPHAN  6491112740659 19-NOV-20
43      43      PRD  1352250453       ORPHAN  6491676585105 23-NOV-20
44      44      PRD  1352250453       ORPHAN  6493754498541 07-DEC-20
45      45      PRD  1352250453       ORPHAN  6505194885552 16-FEB-21
46      46      PRD  1352250453       ORPHAN  6506134480796 21-FEB-21
47      47      PRD  1352250453       ORPHAN  6506147964368 21-FEB-21
48      48      PRD  1352250453       ORPHAN  6506415997452 23-FEB-21
49      49      PRD  1352250453       ORPHAN  6516084726181 21-APR-21
50      50      PRD  1352250453       ORPHAN  6516942842956 26-APR-21
51      51      PRD  1352250453       ORPHAN  6520146079148 23-MAY-21
52      52      PRD  1352250453       ORPHAN  6522049248024 06-JUN-21
53      53      PRD  1352250453       ORPHAN  6529792338188 28-JUL-21
54      54      PRD  1352250453       ORPHAN  6533931540996 22-AUG-21
55      55      PRD  1352250453       ORPHAN  6541654583957 27-SEP-21
56      56      PRD  1352250453       ORPHAN  6547395262155 20-OCT-21
57      57      PRD  1352250453       ORPHAN  6549851396517 31-OCT-21
58      58      PRD  1352250453       ORPHAN  6549859042608 31-OCT-21
59      59      PRD  1352250453       ORPHAN  6554974463446 21-NOV-21
60      60      PRD  1352250453       ORPHAN  6555002692454 21-NOV-21
61      61      PRD  1352250453       ORPHAN  6560037620145 12-DEC-21
62      62      PRD  1352250453       ORPHAN  6572404510789 07-FEB-22
63      63      PRD  1352250453       ORPHAN  6573278923413 10-FEB-22
64      64      PRD  1352250453       ORPHAN  6573612206898 11-FEB-22
65      65      PRD  1352250453       ORPHAN  6574219150557 14-FEB-22
66      66      PRD  1352250453       ORPHAN  6574446599929 15-FEB-22
67      67      PRD  1352250453       ORPHAN  6574733753234 16-FEB-22
68      68      PRD  1352250453       ORPHAN  6575284306973 18-FEB-22
69      69      PRD  1352250453       ORPHAN  6575932360856 21-FEB-22
70      70      PRD  1352250453       ORPHAN  6576184619358 22-FEB-22
71      71      PRD  1352250453       ORPHAN  6576483271333 23-FEB-22
72      72      PRD  1352250453       ORPHAN  6576483271333 23-FEB-22
73      73      PRD  1352250453       ORPHAN  6576488218054 23-FEB-22
74      74      PRD  1352250453       ORPHAN  6576751793528 24-FEB-22
75      75      PRD  1352250453       ORPHAN  6576760598850 24-FEB-22
76      76      PRD  1352250453       ORPHAN  6577014030463 25-FEB-22
77      77      PRD  1352250453       ORPHAN  6577276447034 26-FEB-22
78      78      PRD  1352250453       ORPHAN  6577498266856 27-FEB-22
79      79      PRD  1352250453       ORPHAN  6577655247878 28-FEB-22
80      80      PRD  1352250453       ORPHAN  6577922958840 01-MAR-22
81      81      PRD  1352250453       ORPHAN  6578182486068 02-MAR-22
82      82      PRD  1352250453       ORPHAN  6578472039163 03-MAR-22
83      83      PRD  1352250453       ORPHAN  6578790664228 04-MAR-22
84      84      PRD  1352250453       ORPHAN  6578791447776 04-MAR-22
85      85      PRD  1352250453       ORPHAN  6579600980093 08-MAR-22
86      86      PRD  1352250453       ORPHAN  6580043914718 09-MAR-22
87      87      PRD  1352250453       ORPHAN  6581184664237 15-MAR-22
88      88      PRD  1352250453       ORPHAN  6585156574467 29-MAR-22
89      89      PRD  1352250453       ORPHAN  6585320699260 30-MAR-22
90      90      PRD  1352250453       ORPHAN  6594564672284 17-MAY-22
91      91      PRD  1352250453       ORPHAN  6601496376855 16-JUN-22
92      92      PRD  1352250453       CURRENT 6629485928030 04-OCT-22

There are tons of orphan incarnations found, but there are only 2 PARENT incarnations and 1 CURRENT, definitely the CURRENT incarnation is not working for my DB, so I've to set the DB incarnation to one of the PARENT ones.

Solution:

Reset the incarnation of the database to the most recent "Reset SCN"  PARENT incarnation, from above output incarnation# 2 is the one with most recent "Reset SCN"

RMAN> reset database to incarnation 2;

database reset to incarnation 2


References:

Roll Forward Physical Standby Using RMAN Incremental Backup in Single Command (Doc ID 2431311.1)

Saturday, December 26, 2020

RECOVER MANAGED STANDBY DATABASE INSTANCES ALL is not working on 19.5

 Starting with 12.2, Oracle introduced a new feature called Multi-Instance Redo Apply (MIRA), in a nutshell, this feature allows the Redo Apply to sync the data from all standby nodes (in parallel) which improves the scalability of Redo Apply on Standby RAC DB. But the bad news is that, this feature is not functioning properly, and it's highly advisable to avoid using it!

When I tested this feature on a 19.5 RAC two nodes standby DB I figured out that while the Redo Apply and MRP process is running after enabling MIRA, none of the RAC standby instance are applying the Redo data on the database! Let's have a look into this demonstration:

On Standby node1: Enable MIRA by adding INSTANCES ALL argument to the Recover command]

SYS@drorcl1>  RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT INSTANCES ALL;
Media recovery complete.

[oracle@drhost01]$ ps -ef|grep mrp|grep -v grep
oracle 18492 1 0 14:14 ? 00:00:00 ora_mrp0_drorcl1

Now let's test the data sync on the Standby DB by creating a new table on the Primary DB:

On Primary:

SYS@orcl1> create table test(a number);
Table created.

SYS@orcl1> alter system archive log current;
System altered.

On Standby:

SYS@drorcl1>  desc test

ERROR:
ORA-04043: object test does not exist


The table didn't sync to the standby DB although the MRP process is running!

Once I noticed that MIRA is not working I switched back to the legacy single instance Redo Apply mode:

On Standby:

SYS@orcl1> RECOVER MANAGED STANDBY DATABASE CANCEL;
Media recovery complete.

SYS@orcl1> RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT;
Media recovery complete.

SYS@orcl1> desc test

Name Null? Type
----------------------------------------------------------------------------------------------- --------
A NUMBER

I opened an SR with Oracle Support, and they came back suggesting upgrading the DB to 19.6, then apply Patch 32071879

Considering the effort of upgrading the DB to 19.6 using RU patch then applying the mentioned patch, I decided to take the fastest route and keep using the legacy Redo Apply option!

Moreover, by having a look to the bugs resolved by that patch, you can easily notice that MIRA is an immature or better say a "catastrophic feature!"

😱

Conclusion:

The new feature of Multi-Instance Redo Apply (MIRA) in 19c is unreliable, it's advised to keep using the legacy Single instance Redo Apply option.

Tuesday, December 8, 2020

Standby RAC Database Creation Steps 19c

Creation of a standby database in 19c is pretty simliar to 11g except of two points:

1- Using the compressed feature in the Duplicate command  "USING COMPRESSED BACKUPSET" can speed up the whole process by 200%. [12.2 New feature]

2- TEMPFILES will be created automatically whenStandby DB get opened in OPEN mode.

Before we start please pay attention to the following details of both Primary and Standby environments [Both are RAC environments]
Environment Node Name DB_NAME Instances_Name Public IP Virtual IP (VIP) Private IP
Primary Node1 ORCL_N1 ORCL orcl1 192.168.10.1 Not in Scope Not in Scope
Primary Node2 ORCL_N2 orcl2 192.168.10.2 Not in Scope Not in Scope
Standby Node1 ORCLDR_N1 ORCLDR orcldr1 10.1.10.1 10.1.10.10 10.110.10.1
Standby Node2 ORCLDR_N2 orcldr2 10.1.10.2 10.1.10.11 10.110.10.2

As a matter of fact, Grid Infrastructure 19c should be already installed on both standby nodes, database software installation should be installed as well, you don't have to create a database.


Step 1: Modify the tnsnames.ora file [On both Primary nodes]:

 
This is to add two tnsentries, one for Primary DB "ORCL" and another for the Standby DB "ORCLDR".
# vi /u01/oracle/19c/network/admin/tnsnames.ora

ORCL =
  (DESCRIPTION =
    (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.10.1)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.10.2)(PORT=1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )
  )

ORCLDR=
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.1)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.2)(PORT=1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ORCLDR)
    )
  )


Note: You can also Specify (UR=A) option next to service_name to connect to an instance in NOMOUNT mode: [this can work as an alternative to register the service statically with the listener, but it is not guaranteed to work all the time]
ORCLDR=
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.1)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.2)(PORT=1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ORCLDR)
      (UR=A)
    )
  )



Step 2: Modify listener.ora: [On both Primary nodes]

To statically register the local instance SID:
[On Primary node1]
# vi /u01/grid/19c/network/admin/listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC = (GLOBAL_DBNAME = orcl1) (ORACLE_HOME = /u01/oracle/19c) (SID_NAME = orcl1))
  )


# lsnrctl reload listener

[On Primary node2]
# vi /u01/grid/19c/network/admin/listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC = (GLOBAL_DBNAME = orcl2) (ORACLE_HOME = /u01/oracle/19c) (SID_NAME = orcl2))
  )

# lsnrctl reload listener


Step 3: Modify tnsnames.ora  [On both Standby nodes]

This is to add two tnsentries, one for Primary DB "ORCL" and another for the Standby DB "ORCLDR".
# vi /u01/oracle/19c/network/admin/tnsnames.ora

ORCL =
  (DESCRIPTION =
    (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.10.1)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.10.2)(PORT=1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )
  )

ORCLDR=
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.1)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.2)(PORT=1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ORCLDR)
    )
  )


Step 4: Modify listener.ora:  [On Standby node1]

Statically register the local instance SID:
# vi /u01/grid/19c/network/admin/listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC = (GLOBAL_DBNAME = orcldr) (ORACLE_HOME = /u01/oracle/19c) (SID_NAME = orcldr1))
  )


# lsnrctl reload listener


* Do a TNSPING from both Primary and Standby nodes and make sure it's working.
# tnsping ORCL

# tnsping ORCLDR

Step 5: Make sure the Primary DB is in ARCHIVELOG & FORCE LOGGING modes: [On Primary DB]

SQL> ARCHIVE LOG LIST;
SQL> ALTER DATABASE FORCE LOGGING;
SQL> ALTER DATABASE ARCHIVE LOG CURRENT;

Make sure below 2 parameters are well configured to reflict the database name: [db_unique name on Primary DB should be orcl and on Standby DB it will  be orcldr]
SQL> show parameter db_unique_name
SQL> show parameter db_name

Step 6: Log Shipping Setup: [On Primary DB]

-- Here you should use the DB_UNIQUE_NAME of each of primary and standby:
SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(orcl,orcldr)';
SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=orcldr NOAFFIRM LGWR ASYNC delay=0 VALID_FOR=(all_logfiles,PRIMARY_ROLE) max_failure=0 reopen=300 net_timeout=90 DB_UNIQUE_NAME=orcldr';

-- Here you should use service names which you already configured earlier in tnsnames.ora:
SQL> ALTER SYSTEM SET log_archive_dest_state_2=enable;
SQL> ALTER SYSTEM SET remote_login_passwordfile=exclusive SCOPE=SPFILE;
SQL> ALTER SYSTEM SET fal_server='orcldr';
SQL> ALTER SYSTEM SET fal_client='orcl';
SQL> ALTER SYSTEM SET standby_file_management=auto;

Step 7: Create Standby REDOLOG Groups [On Primary DB]:

Standby REDOLOG groups should be same number as regular REDOLOG groups +1 for each thread. e.g. if the primary database has 5 REDOLOG groups for each thread, then I should create 6 Standby REDOLOG groups for each thread.

SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 1
     GROUP 10 ('+REDO1') SIZE 1g,
     GROUP 11 ('+REDO1') SIZE 1g,
     GROUP 12 ('+REDO1') SIZE 1g,
     GROUP 13 ('+REDO1') SIZE 1g,
     GROUP 14 ('+REDO1') SIZE 1g,
     GROUP 15 ('+REDO1') SIZE 1g;



SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 2
     GROUP 16 ('+REDO2') SIZE 1g,
     GROUP 17 ('+REDO2') SIZE 1g,
     GROUP 18 ('+REDO2') SIZE 1g,
     GROUP 19 ('+REDO2') SIZE 1g,
     GROUP 20 ('+REDO2') SIZE 1g,
     GROUP 21 ('+REDO2') SIZE 1g;
    

Step 8: Make sure the SNAPSHOT controlfile directory is exist and writable on both Primary nodes: [Even if the location is not shared between both nodes!]

# rman
RMAN> show all;
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/acfs/backup/sprint/snapcf_sprint.f';

Step 9: Create the password file on [Primary node1]

# orapwd file=/u01/oracle/19c/dbs/orapworcl1 password=Pass#0rd entries=10


If the Password file is already exist on ASM, then Copy it to $ORACLE_HOME/dbs:

[To make it seamless reset the SYS password on both Primary Instances]

SQL> alter user sys identified by "Pass#0rd";


# srvctl config database -d orcl | grep pwd
+DATA/ORCL/PASSWORD/pwdorcl.256.1039797083

# asmcmd cp +DATA/ORCL/PASSWORD/pwdorcl.256.1039797083 $ORACLE_HOME/dbs/orapworcl1

Copy the password file to [Primary Node2]:
# scp -p orapworcl1 oracle@192.168.10.2:/u01/oracle/19c/dbs/orapworcl2

Copy the password file to [Standby Node1]
# scp -p orapworcl1 oracle@10.1.10.1:/u01/oracle/19c/dbs/orapworcldr1

Copy the password file to [Standby Node2]
# scp -p orapworcl1 oracle@10.1.10.2:/u01/oracle/19c/dbs/orapworcldr2


Step 10: Create the pfile on Target: [on Standby node1]

# vi $ORACLE_HOME/dbs/initorcldr1.ora

*.sga_target=5g
*.pga_aggregate_target=2g
*.control_files=+DATA,+REDO1,+REDO2
*.db_create_file_dest='+DATA'
*.db_create_online_log_dest_1='+REDO1'
*.db_create_online_log_dest_2='+REDO2'
*.db_name='orcldr'
*.db_recovery_file_dest='+RECO'
*.db_recovery_file_dest_size=900g
*.db_unique_name='orcldr'
*.remote_login_passwordfile='exclusive'



Step 11: Create Directory Structure: [Similar to what to be configured on PFILE and DBA_DIRECTORIES] [on both Standby nodes]

[Standby node1]
# mkdir -p /u01/oracle/admin/orcldr/adump
# mkdir -p /u01/oracle/diag/rdbms/orcldr1

[Standby node2]
# mkdir -p /u01/oracle/admin/orcldr/adump
# mkdir -p /u01/oracle/diag/rdbms/orcldr2

Step 12: Start the new orcldr1 instance in nomount mode with pfile: [on Standby nodes1]

# cd $ORACLE_HOME/dbs
# export ORACLE_SID=orcldr1
# sqlplus "/ as sysdba"
SQL> startup nomount pfile=initorcldr1.ora

Step 13: Test the connection between Primary and Standby instances:

-- From [Primary node1] try to connect to the standby instance:
SQL> conn sys/Pass#0rd@orcldr as sysdba
connected

-- From [Primary node2] try to connect to the standby instance:
SQL> conn sys/Pass#0rd@orcldr as sysdba
connected

-- From [Standby node1] try to connect to the Primary DB:
SQL> conn sys/Pass#0rd@orcl as sysdba
connected

 
Step 14: Create the RMAN Duplicate script: [On Primary node1]
 
# vi ~/duplicate_orcldr_standby.rman

run {
allocate channel disk1 type disk;
allocate channel disk2 type disk;
allocate channel disk3 type disk;
allocate auxiliary channel aux1 type disk;
allocate auxiliary channel aux2 type disk;
allocate auxiliary channel aux3 type disk;
duplicate target database for standby from active database USING COMPRESSED BACKUPSET
spfile
parameter_value_convert 'ORCL','ORCLDR'
set db_name='orcl'
set db_unique_name='orcldr'
set cluster_database='FALSE'
set control_files='+DATA','+REDO1','+REDO2'
set db_create_file_dest='+DATA'
set db_create_online_log_dest_1='+REDO1'
set db_create_online_log_dest_2='+REDO2'
set db_name='orcl'
set db_recovery_file_dest='+RECO'
set db_recovery_file_dest_size='900G'
set sga_target='5g'
set db_cache_size='2g'
set shared_pool_size='1g'
set pga_aggregate_target='2g'
set remote_login_passwordfile='exclusive'
set instance_number='1'
set audit_file_dest='/u01/oracle/admin/orcldr/adump'
set diagnostic_dest='/u01/oracle'
set db_file_name_convert='+DATA/orcl/datafile','+DATA/orcldr/datafile','+DATA/orcl/tempfile','+DATA/orcldr/tempfile'
set log_file_name_convert='+REDO1/orcl/onlinelog','+REDO1/orcldr/onlinelog','+REDO2/orcl/onlinelog','+REDO2/orcldr/onlinelog'
set db_create_file_dest='+DATA'
set fal_client='orcldr'
set fal_server='orcl'
set standby_file_management='AUTO'
set log_archive_config='dg_config=(orcldr,orcl)'
set log_archive_dest_2='SERVICE=orcl NOAFFIRM LGWR ASYNC delay=0 VALID_FOR=(all_logfiles,PRIMARY_ROLE) max_failure=0 reopen=300 net_timeout=90 DB_UNIQUE_NAME=orcl';
}


Note: Using COMPRESSION with duplicate command will speed up the duplicate by 200% in average]

Step 15: Run the Duplicate operation: [From Primary node1]

Note: Don't connect "TARGET /" nor "AUXILIARY /", you have to connect using ID and password and SERVICE NAME i.e: sys/Pass#0rd@ORCLDR

# export ORACLE_SID=orcl1
# nohup $ORACLE_HOME/bin/rman target sys/Pass#0rd@ORCL auxiliary sys/Pass#0rd@ORCLDR cmdfile=~/duplicate_orcldr_standby.rman | tee ~/duplicate_standby.log 2>&1 &

Step 16:  Recover the Standby DB: [on Standby node1]

SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP

Note: The database will not open unless it apply all the pending archivelogs. [This happens AUTOMATICALLY]

Step 17: Disable log_archive_dest_state_2 & 3: [on Standby DB]

SQL> ALTER SYSTEM SET log_archive_dest_state_2=defer;
SQL> ALTER SYSTEM SET log_archive_dest_state_3=defer;
SQL> ALTER SYSTEM SET dispatchers = "(PROTOCOL=TCP) (SERVICE=orcldrXDB)";

Step 18: Create Standby REDOLOG Groups [on Standby DB]:

Same number of Standby REDOLOG groups on Primary should be created on Standby DB as well:

SQL> RECOVER MANAGED STANDBY DATABASE CANCEL;

SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 1
     GROUP 10 ('+REDO1') SIZE 1g,
     GROUP 11 ('+REDO1') SIZE 1g,
     GROUP 12 ('+REDO1') SIZE 1g,
     GROUP 13 ('+REDO1') SIZE 1g,
     GROUP 14 ('+REDO1') SIZE 1g,
     GROUP 15 ('+REDO1') SIZE 1g;


SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 2
     GROUP 16 ('+REDO2') SIZE 1g,
     GROUP 17 ('+REDO2') SIZE 1g,
     GROUP 18 ('+REDO2') SIZE 1g,
     GROUP 19 ('+REDO2') SIZE 1g,
     GROUP 20 ('+REDO2') SIZE 1g,
     GROUP 21 ('+REDO2') SIZE 1g;

    
Step 19: Turn ON Flashback [On Standby DB]:

SQL> ALTER DATABASE FLASHBACK ON;
SQL> RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT;

Note: "USING CURRENT LOGFILE" in the Recover command is not required anymore in 19c.

Step 20: Restart log_archive_dest_state_2 [On Primary DB]:
This to refresh the redo log shipping process on promary DB and fix any gap.

SQL> ALTER SYSTEM SET log_archive_dest_state_2=defer;
SQL> ALTER SYSTEM SET log_archive_dest_state_2=enable;

Step 21: Start the Recovery process [On Standby DB]:

SQL> RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT;

Step 22: Fix RMAN Settings [On Standby DB]:

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;
RMAN> CONFIGURE BACKUP OPTIMIZATION ON;
RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;
RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/acfs/backup/ORCLDR/%F';
RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'SBT_TAPE' TO '%F';
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/acfs/backup/ORCLDR/snapcf_orcldr.f';

[On both Standby Nodes] Create the full path for the CONTROLFILE SNAPSHOT BACKUP: [It should be a shared location between both nodes, but if this not possible for any reason just create it locally on both nodes with the right permissions]
# mkdir -p /acfs/backup/ORCLDR
# chown oracle:oinstall -R /acfs/backup

Step 23: Configure the archivelog deletion policy [On Primary DB]:

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

Step 24: Convert the standby to RAC DB [On Standby node1]:

SQL>ALTER SYSTEM SET cluster_database=true SCOPE=SPFILE;
    ALTER SYSTEM SET cluster_database_instances=2 SCOPE=SPFILE SID='*';
    ALTER SYSTEM SET instance_number=1         SCOPE=SPFILE SID='orcldr1';
    ALTER SYSTEM SET instance_number=2           SCOPE=SPFILE SID='orcldr2';
    ALTER SYSTEM SET thread=1                  SCOPE=SPFILE SID='orcldr1';
    ALTER SYSTEM SET thread=2                  SCOPE=SPFILE SID='orcldr2';
    ALTER SYSTEM SET undo_tablespace=UNDOTBS1  SCOPE=SPFILE SID='orcldr1';
    ALTER SYSTEM SET undo_tablespace=UNDOTBS2  SCOPE=SPFILE SID='orcldr2';
    ALTER SYSTEM SET cluster_interconnects='10.110.10.1' SCOPE=SPFILE SID='orcldr1';
    ALTER SYSTEM SET cluster_interconnects='10.110.10.2' SCOPE=SPFILE SID='orcldr2';
    ALTER SYSTEM SET local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.10)(PORT=1521))' sid='orcldr1';
    ALTER SYSTEM SET local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.1.10.11)(PORT=1521))' sid='orcldr2';


Create the SPFILE on a shared location between Standby RAC Nodes which is +DATA:
SQL> create pfile='/u01/oracle/19c/dbs/initorcldr.ora' from spfile;
SQL> create spfile='+DATA' from pfile='/u01/oracle/19c/dbs/initorcldr.ora';

Get the spfile full path on ASM using asmcmd to use it in the next step:
# asmcmd ls -lst +DATA/ORCLDR/PARAMETERFILE
+DATA/ORCLDR/PARAMETERFILE/spfile.345.1040975313

Register the database with clusterware services:
# srvctl add database -db orcldr -oraclehome /u01/oracle/19c -dbtype RAC -spfile +DATA/ORCLDR/PARAMETERFILE/spfile.345.1040975313 \
-role PHYSICAL_STANDBY -startoption open -stopoption immediate -diskgroup "DATA,REDO1,REDO2,RECO" -policy AUTOMATIC -verbose


Register the instances with clusterware services:
# srvctl add instance -db orcldr -instance orcldr1 -node ORCLDR_N1
# srvctl add instance -db orcldr -instance orcldr2 -node ORCLDR_N2

Shutdown the DB [on Standby node1]:
SQL>  shutdown immediate;

Test Startup/Stop of the Standby DB [On Both Standby nodes]
# srvctl start database -db orcldr
# srvctl stop  database -db orcldr
# srvctl start database -db orcldr

*Make sure the Redo log files is being shipped from Primary to Standby whenever any of the standby instances are down.

Start the Recovery Process: [On any of Standby nodes]
SQL> RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT;

Lastly: You can use the following scripts to complete your setup:

Configure LAG monitoring

Setup Archive deletion script

In case you have a need to create many standby DBs, this script can help you out: https://github.com/asiandevs/PhysicalStandbyWithDGBroker

Wednesday, May 20, 2020

Duplicate of a Standby DB fails with ORA-19845

While creating a Standby database on a RAC environment from another RAC database using duplicate method I was getting this weird error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 05/20/2020 14:20:14
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19845: error in backupControlFile while communicating with remote database server
ORA-17628: Oracle error 1580 returned by remote Oracle server
ORA-01580: error creating control backup file
ORA-19660: some files in the backup set could not be verified
ORA-19661: datafile 0 could not be verified
ORA-19845: error in backupControlFile while communicating with remote database server
ORA-17628: Oracle error 1580 returned by remote Oracle server
ORA-01580: error creating control backup file


I was running the duplicate command from the primary DB (Node1)

After long investigation I figured out that the Snapshot controlfile location is not shared between both RAC instances (on the primary side):

RMAN> show SNAPSHOT CONTROLFILE NAME;

RMAN configuration parameters for database with db_unique_name SPRINTS are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/cloudfs/backup/sprint/snapcf_sprint.f';


That location "/cloudfs/backup/sprint" was only available on Node1 and not available on Node2

I've simply gone to Node2 and created the full location path and the duplicate succeeded!
On Node2:

# mkdir -p /cloudfs/backup/sprint
# chown oracle /cloudfs/ -R


Conclusion:
As a rule of thumb, the SNAPSHOT CONTROLFILE location should be available on all RAC nodes. Even if it's not shared between all nodes, it just needs to be exist!


Wednesday, November 1, 2017

Shell Script To Monitor/Check The Lag/Sync Status Between Primary And Standby Oracle Database

Script Description:
This Linux shell script runs from the PRIMARY DB server to monitor and checks the lag/synchronization status between Primary and Standby Oracle DBs and reports the lag [if happened] to the user.

Download the script:
https://www.dropbox.com/s/a9k3rn928vj3ooc/check_standby_lag.sh?dl=0

How to use the script:
 
First: On the Primary DB: Create a DBA user to be used inside the script to check the dataguard status:
e.g. This is just n example, you can create the user with a proper name and password.
 
SQL> create user DBA_USER identified by "ABC123" profile;
SQL> grant dba to
DBA_USER;
 

Second: Unfortunately, this script will not be able to know what is the primary, standby database names itself; so you must manually modify the variables' values in orange color in the Variables section to match your environment: [Those variables are at the begging of the script]

# ######################################
# Variables MUST be modified by the user: [Otherwise the script will not work]
# ######################################

# Here you replace youremail@yourcompany.com with your Email address:
EMAIL="youremail@yourcompany.com"

# Replace ${ORACLE_SID} with the Primary DB instance SID:
ORACLE_SID=${ORACLE_SID}

# Replace STANDBY_TNS_ENTRY with the Standby Instance TNS entry you configured in the primary site tnsnames.ora file: e.g. orcl_standby
DRDBNAME=STANDBY_DB

# Replace ${ORACLE_HOME} with the ORACLE_HOME path on the primary server:
ORACLE_HOME=${ORACLE_HOME}

# Log Directory Location:
LOG_DIR='/tmp'
 
# Here you replace DBA_USER with a real user having DBA privilege:
ID=DBA_USER

# Here you replace ABC123 with the DBA user password on the standby DB:
CRD='ABC123'

# Replace "5" with the number of LAGGED ARCHIVELOGS if reached an Email alert will be sent to the receiver:
LAGTHRESHOLD=5


You can schedule this script in the crontab to run e.g. every 15 minutes to get you alerted if the number of lagged archive logs cross the predefined threshold.

Reminder: This script runs from the primary DB server NOT from the standby server.
Hope you will find it useful.

This script is part of DBA Bundle scripts: http://dba-tips.blogspot.ae/2014/02/oracle-database-administration-scripts.html

Tuesday, January 17, 2017

Script to delete applied archivelogs on a STANDBY database

This script is designed to run against STANDBY databases to deleted applied archives older than specific number of hours defined by the user. It maintains that un-applied archives remain un-touched.

Download the script:
https://www.dropbox.com/s/xx5o6b94rv5ps30/delete_applied_archives_on_standby.sh?dl=0


How to use the script:

⏩ Set MAIL_LIST variable to your E-mail to receive an email alert if archives are not applied. 
    e.g. MAIL_LIST="john.smith@abc.com"

Specify the candidate archives for deletion older than N hours by setting LAST_N_HOURS variable to the number of hours.

    e.g. Deleting applied archives older than 24 hours:
     LAST_N_HOURS=24

⏩ You can EXCLUDE any instance from having the script to run against by passing the INSTANCE_NAME you want to exclude to EXL_DB variable.

e.g. excluding orcl from archive deletion:

EXL_DB="\-MGMTDB|ASM|orcl"


⏩ You can use FORCE option when deleting the archives from RMAN console: [Y|N] [Default is NO]
    e.g. FORCE_DELETION=Y

⏩ You can decide to CROSSCHECK the archivelogs after the archivelogs deletion: [Y|N] [Default is YES]
    e.g. VALIDATE_ARCHIVES=Y

⏩ Also you can schedule it to run periodically in the Oracle's user crontab as well.

DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".

Sunday, December 18, 2016

Managed Recovery Stopped Applying Archives on Standby Database After Enabling Apply Delay on The Primary Archive Process

I had a requirement to set the archive apply delay feature between the primary and the standby DB, where the archives will be sent immediately to the standby site but will get applied in a delay of one hour. Such feature enables you to eliminate any erroneous update get executed on the primary from being applied immediately on the standby DB,which gives the DBA some time to think whether to fix the logical corruption or failing over to the standby DB.

Back to my story, On the primary DB, I've set the delay on the archival process that shift the archives to the standby DB to 60 minutes:

SQL> alter system set log_archive_dest_6='service="testprds"','LGWR ASYNC NOAFFIRM delay=60 optional compression=enable max_failure=0 max_connections=1 reopen=300 db_unique_name="testprds" net_timeout=90','valid_for=(all_logfiles,primary_role)';

Restart the archival process: [To let the changes take effect]
SQL> alter system set log_archive_dest_state_6=DEFER;
SQL> alter system set log_archive_dest_state_6=ENABLE;

Restart the recover on the standby which was already started in nodelay mode to start it in delayed mode:
SQL> recover managed standby database cancel;
SQL> recover managed standby database parallel 16 DISCONNECT;

Now I can see in the standby alertlog archives are getting received immediately on the standby and acknowledged with 60 minutes delay for apply:

Archived Log entry 378319 added for thread 1 sequence 681301 ID 0x6a29c0d6 dest 1:
ARCs: Archive log thread 1 sequence 681301 available in 60 minute(s)

Up to here, so far so good, it works as expected, but the horror didn't start yet :-) 
After 1 hour. There are no archives applied on the standby DB.
After 2 hours. Same situation, non of archives applied since I modified the archive process with delay option ! Something went wrong.

Looks easy! Not a rocket science, stop/start the recovery process with NODELAY option and the recovery will start overriding the delay:

SQL> RECOVER MANAGED STANDBY DATABASE cancel;
Media recovery complete.

SQL> RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE NODELAY DISCONNECT;
ORA-01153: an incompatible media recovery is active

WHAT?! It was just successfully cancelled !

Hopelessly I tried the following troubleshooting techniques, but non of them work:
- Repeating same above commands didn't help. 
- No MRP process was found on OS side to kill.
- Altered the delay parameter back to 0 and restart (defer/enable) the archive process on the primary DB didn't help.
- Killing ora_arc* processes on the standby didn't help.
- Deleting the archives that were marked with 60 minutes delay on standby, and recopy those archives again from primary site did help as well.

Now, it's time to think logically. This error means that although you've cancelled the media recovery there is still another media recovery processes is running didn't get cancelled when you executed "recover managed standby database cancel" command. Is this can happen? Yes it can happen !

So, we need to look into all recovery processes and their statuses:

SQL> select PROCESS,PID,STATUS from v$managed_standby;

PROCESS PID         STATUS
-------------   ----------  ------------
RFS            1292        IDLE
RFS            1095        IDLE
MR(fg)          19509       WAIT_FOR_LOG

Yes !, it's the one showing WAIT_FOR_LOG status.

Killed it from Linux shell using its PID:

# kill -9 19509

Now, starting media recovery on standby DB with NODELAY option:

SQL> RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE NODELAY DISCONNECT;
Media recovery complete.

Phew, finally the managed recovery is running!

Someone may ask why you didn't just restart the standby DB instance and it will force clear any stale recovery process?
The answer, imagine you are using your standby DB in Active Data Guard mode to support read only/reporting applications, restarting the standby database will impact the availability of those applications.

Sunday, August 17, 2008

Standby Database Switchover | Failover

Switch Over:

Switch over means to switch the roles between the Primary DB and the Standby DB, so after the Switch over the Primary DB will act as the standby DB, and the Standby DB will act as the Primary DB.
This approach is usually being used during the DR drills, where the standby database will be opened in READ/WRITE mode, to allow production applications to use it during the drill. Once the drill is done, the DBA can switch back the roles between both sites to get back to the original status, without the need to recreate any of the primary or standby databases.

Before the switch over: Do the following checking
---------------------------


1= Verify that the last sequence# received and the last sequence# already applied on the standby database:


SQL> select max(al.sequence#) "Last Received",max(lh.sequence#) "Last Applied" from v$archived_log al, v$log_history lh;

If the two numbers are the same then the standby has applied all the archives been sent from the primary site.

If not, follow these steps:
------------------------------

2= Verify that the standby is in mount state:

SQL> select open_mode from v$database;

3= Determine if there is any archive gap on the physical standby database:

SQL> select * from v$archive_gap;

4= Verify that the managed recovery process is running:

SQL> select process,status from v$managed_standby;

-If you do not see MRP process this mean the managed recovery not working so start it by:

SQL> recover managed standby database disconnect;



On the Primary DB:
-----------------------
SQL> select switchover_status from v$database;

STATUS should be ‘SESSIONS ACTIVE’ or ‘TO STANDBY’

On the Standby DB:
-----------------------
SQL> select switchover_status from v$database;

STATUS should be ‘SESSIONS ACTIVE’ or ‘TO PRIMARY’


Switch over Steps:   
=============
1- Apply all pending archives on the Standby DB:

-   --------------------------------------------------------

>Disconnect all sessions.
>Stop REDO apply:
> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE NODELAY;
> For RAC:
> Stop all instance except single instance.
> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE; --Do not wait for the network.


2-  On the Primary DB:

-    ------------------------
> Stop all RAC instance except one instance:
> Disconnect all sessions...

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY with session shutdown;

3-  On the Standby DB:
-    ------------------------
=> Ensure that all archives has been applied.

=> Ensure there is no connected sessions on the Standby DB.....

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;

4-  On the old Primary DB: (New Standby)
-    ----------------------------
Disable all archive processes that send the archives to the old standby:
SQL> ALTER SYSTEM SET log_archive_dest_state_2='DEFER' SCOPE=BOTH;

* Check if any of the other archive dest parameters is enabled e.g. log_archive_dest_state_3 ,4 ,5 ... 

5-  On the old Standby DB: (New Primary)
-    -----------------------------
Enable the log_archive_dest_2 to transfer the archives to the old primary (new standby):

SQL> Alter system set log_archive_dest_2='service="PRIMARY" db_unique_name=PRIMARY compression=enable' scope=both;
SQL> ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;

SQL> Select DATABASE_ROLE,OPEN_MODE,PROTECTION_MODE from v$database;

DATABASE_ROLE OPEN_MODE  PROTECTION_MODE
---------------- ---------- --------------------
PRIMARY MOUNTED    MAXIMUM PERFORMANCE

SQL> ALTER DATABASE OPEN;

SQL> Alter system set standby_file_management='AUTO';


6-  On the old Primary DB: (New Standby)

-    ----------------------------
Restart the database in mount mode to act as a standby DB and start to apply the shipped archives from the old standby (new primary).

SQL> SHUTDOWN IMMEDIATE;
SQL> startup NOMOUNT;
SQL> alter database MOUNT standby database;
SQL> recover managed standby database disconnect;


7-  On the old Standby DB: (New Primary)
-    -----------------------------

- If it's a RAC setup, start up all instances.
- Modify the connection string inside application config files to point to the old standby DB (new primary)


Switch Back:  From New Primary to the new DR
=========
On old DR:
-----------
SQL> ALTER SYSTEM SWITCH LOGFILE;

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;

On old Primary: 
---------------
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;

On old DR: Disbale sending archives to old Primary
-----------
ALTER SYSTEM SET log_archive_dest_state_2='DEFER' SCOPE=BOTH;

On old Primary: 
---------------
SQL> ALTER SYSTEM SET log_archive_dest_state_1='ENABLE' SCOPE=BOTH;
SQL> Select DATABASE_ROLE,OPEN_MODE,PROTECTION_MODE from v$database;

DATABASE_ROLE OPEN_MODE  PROTECTION_MODE
---------------- ---------- --------------------
PRIMARY MOUNTED    MAXIMUM PERFORMANCE

SQL> ALTER DATABASE OPEN;

On DR:
-------
SQL>SHUTDOWN IMMEDIATE;
SQL>startup NOMOUNT;
SQL>alter database MOUNT standby database;
SQL>recover managed standby database disconnect;
SQL>Select DATABASE_ROLE,OPEN_MODE,PROTECTION_MODE from v$database;


Fail Over:

The FAIL OVER approach is used in the real disaster scenarios where the primary site is not exist/accessible anymore, because of flood, earthquake, war, power failure, or whatever the catastrophe caused in the primary site.

All the failover steps will be done on the DR site.

Normal Fail over:
============
On the Standby DB:
------------------------
Terminate the managed recovery mode:
-----------------------------------------------
SQL>recover managed standby database nodelay;

SQL>alter database recover managed standby database finish;

Open the Standby database in READ/WRITE mode to act as a primary DB:
-------------------------------------------------------------------------------------------
SQL>alter database commit to switchover to primary;

SQL>alter database open;


Forced Fail over:
============
SQL>alter database recover managed standby database finish force;

SQL>alter database activate standby database [skip standby logfile];

SQL>shutdown immediate;

SQL>startup mount;

SQL>alter database open resetlogs;