This example illustrates how to backup and recover a 2-node, 10gR2 RAC cluster on Linux. Backup and recovery operations for RAC databases are similar to that of single instance databases, however there are some key differences – the goal of this post is to highlight the steps required for recoverying RAC databases.
Backup the DatabaseUse the method you prefer. I used a NOCATALOG, disk-based backup for this example, using the scripts foundSimulate the Failure
Shutdown the database and simulate a failure by dropping and/or corrupting a datafile.
# export ORACLE_SID=em1 # . oraenv # srvctl stop database -d em # srvctl status database -d em Instance em1 is not running on node rac1 Instance em2 is not running on node rac2
# cd /u02/oracle/ora1/em
# rm sysaux01.dbf
Verify the Failure
# srvctl start instance -d em -i em1 PRKP-1001 : Error starting instance em1 on node rac1 CRS-0215: Could not start resource 'ora.em.em1.inst'.
You will find something similar to the following in the instance's alert log: Sat May 24 16:53:47 2008 Errors in file /u02/oracle/logs/em/bdump/em1_dbw0_16947.trc: ORA-01157: cannot identify/lock data file 3 - see DBWR trace file ORA-01110: data file 3: '/u02/oracle/ora1/em/sysaux01.dbf' ORA-27037: unable to obtain file status Linux Error: 2: No such file or directory Additional information: 3 ORA-1157 signalled during: ALTER DATABASE OPEN... Sat May 24 16:53:49 2008 Shutting down instance (abort)
Restore the Database
First, take the database out of cluster mode via:
# sqlplus "/ as sysdba" SQL*Plus: Release 10.2.0.3.0 - Production on Sat May 24 17:02:17 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to an idle instance. SQL> startup nomount; ORACLE instance started. Total System Global Area 268435456 bytes Fixed Size 1261300 bytes Variable Size 251658508 bytes Database Buffers 8388608 bytes Redo Buffers 7127040 bytes SQL> alter system set cluster_database=false scope=spfile sid='*'; System altered. SQL> shutdown immediate; ORA-01507: database not mounted ORACLE instance shut down.
Then restore the database via RMAN:
# rman target=/ Recovery Manager: Release 10.2.0.3.0 - Production on Sat May 24 17:04:10 2008 Copyright (c) 1982, 2005, Oracle. All rights reserved. connected to target database (not started) RMAN> startup mount; Oracle instance started database mounted Total System Global Area 268435456 bytes Fixed Size 1261300 bytes Variable Size 251658508 bytes Database Buffers 8388608 bytes Redo Buffers 7127040 bytes RMAN> restore database; Starting restore at 24-MAY-08 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: sid=154 devtype=DISK channel ORA_DISK_1: starting datafile backupset restore channel ORA_DISK_1: specifying datafile(s) to restore from backup set restoring datafile 00001 to /u02/oracle/ora1/em/system01.dbf restoring datafile 00002 to /u02/oracle/ora1/em/undotbs01.dbf restoring datafile 00003 to /u02/oracle/ora1/em/sysaux01.dbf restoring datafile 00004 to /u02/oracle/ora1/em/users01.dbf restoring datafile 00005 to /u02/oracle/ora1/em/example01.dbf restoring datafile 00006 to /u02/oracle/ora1/em/undotbs02.dbf restoring datafile 00007 to /u02/oracle/ora2/em/mgmt.dbf restoring datafile 00008 to /u02/oracle/ora1/em/mgmt_ecm_depot1.dbf channel ORA_DISK_1: reading from backup piece /u02/oracle/ora3/em/backups/0ijh6j4t_1_1 channel ORA_DISK_1: restored backup piece 1 piece handle=/u02/oracle/ora3/em/backups/0ijh6j4t_1_1 tag=FHB_EM1200805241630 channel ORA_DISK_1: restore complete, elapsed time: 00:06:36 Finished restore at 24-MAY-08
Since my database is so small, I will simply restore the entire thing, however, since you know which datafile is missing, you could simply restore it and then recover the database as necessary.
Recover the Database
RMAN> recover database; Starting recover at 24-MAY-08 using channel ORA_DISK_1 starting media recovery media recovery complete, elapsed time: 00:00:03 Finished recover at 24-MAY-08 RMAN> alter database open; database opened RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows ORACLE error from target database: ORA-06502: PL/SQL: numeric or value error: character to number conversion error
Afterwards, place the database back into cluster mode and startup both instances:
# sqlplus "/ as sysdba" SQL*Plus: Release 10.2.0.3.0 - Production on Sat May 24 17:16:36 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> alter system set cluster_database=true scope=spfile sid='*'; System altered. SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. # srvctl start database -d em [oracle@rac1 bdump]$ srvctl status database -d em Instance em1 is running on node rac1 Instance em2 is running on node rac2