We got the below error , while doing the backup based rman duplicate. RMAN command for duplicate is below.
RUN{
set archivelog destination to '/${target}_archredo1/redologs/';
allocate auxiliary channel ch01 device type 'SBT_TAPE' parms 'ENV=(NB_ORA_CLIENT=$client,BLKSIZE=1048576)' maxopenfiles 1;
allocate channel aux1 device type disk;
duplicate target database to ${target}
UNTIL TIME "to_date('$until_time','mm-dd-yyyy hh24:mi:ss')"
pfile = /oracle/app/oracle/admin/$target/pfile/init$target.ora;
}
Recovery part of the duplication started failing with below error
RMAN-03015: error occurred in stored script Memory Script
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/db_name_archredo1/redologs/arch0000079905_1_822846244.log'
ORA-10877: error signaled in parallel recovery slave
When I checked the alert log of the destination database , we found below error
alter database recover logfile '/db_name_archredo1/redologs/arch0000079905_1_822846244.log'
88982 Media Recovery Log /db_name_archredo1/redologs/arch0000079905_1_822846244.log
88983 Wed May 20 12:35:04 2015
88984 Recovery interrupted!
88985 Recovered data files to a consistent state at change 11114962070940
88986 Media Recovery failed with error 19755
88987 Errors in file /oracle/app/oracle/admin/db_name/diag/rdbms/db_name/db_name/trace/db_name_pr00_146224.trc:
88988 ORA-00283: recovery session canceled due to errors
88989 ORA-19755: could not open change tracking file
88990 ORA-19750: change tracking file: '/dwprod1_db01/oracle/rman_change_trck_dwprod1.dbf'
88991 ORA-27037: unable to obtain file status
88992 Linux-x86_64 Error: 2: No such file or directory
88993 Additional information: 3
88994 Slave exiting with ORA-283 exception
88995 Errors in file /oracle/app/oracle/admin/db_name/diag/rdbms/db_name/db_name/trace/db_name_pr00_146224.trc:
88996 ORA-00283: recovery session canceled due to errors
88997 ORA-19755: could not open change tracking file
88998 ORA-19750: change tracking file: '/dwprod1_db01/oracle/rman_change_trck_dwprod1.dbf'
88999 ORA-27037: unable to obtain file status
89000 Linux-x86_64 Error: 2: No such file or directory
89001 Additional information: 3
89002 ORA-10877 signalled during: alter database recover logfile '/db_name_archredo1/redologs/arch0000079905_1_822846244.log'...
This suggested that RMAN was looking for the Block change tracking file to perform the recovery. Since the path of the BCT file was not present in destination, I decided to disable the change tracking at source itself.
SQL>alter database disable block change tracking;
Database altered.
That did not help , even after disabling the change tracking at source did not help .Since my recover time was some time in the past , the controlfile which was restore by RMAN was always looking for change tracking file.
Then I tried to create a dummy file with same name as source change tracking file , but the recovery was failing again, with below message
Media Recovery failed with error 19755
91851 Errors in file /oracle/app/oracle/admin/db_name/diag/rdbms/db_name/db_name/trace/db_name_pr00_38701.trc:
91852 ORA-00283: recovery session canceled due to errors
91853 ORA-19755: could not open change tracking file
91854 ORA-19750: change tracking file: '/dwprod1_db01/oracle/rman_change_trck_dwprod1.dbf'
91855 ORA-27047: unable to read the header block of file
91856 Linux-x86_64 Error: 25: Inappropriate ioctl for device
91857 Additional information: 1
91858 Slave exiting with ORA-283 exception
91859 Errors in file /oracle/app/oracle/admin/db_name/diag/rdbms/db_name/db_name/trace/db_name_pr00_38701.trc:
91860 ORA-00283: recovery session canceled due to errors
91861 ORA-19755: could not open change tracking file
91862 ORA-19750: change tracking file: '/dwprod1_db01/oracle/rman_change_trck_dwprod1.dbf'
91863 ORA-27047: unable to read the header block of file
91864 Linux-x86_64 Error: 25: Inappropriate ioctl for device
91865 Additional information: 1
91866 ORA-10877 signalled during: alter database recover logfile '/db_name_archredo1/redologs/arch0000079905_1_822846244.log'...
Then I realized that, RMAN needs BCT file for restore , I re enabled change tracking at source
SQL> alter database enable block change tracking using FILE '/source_db01/oracle/rman_change_trck_dwprod1.dbf';
Database altered.
Manually copied the file to destination to same path and started the recovery again . This did the trick and recovery started.
** Note 1 : ** Please include location of BCT file in DB_FILE_NAME_CONVERT parameter as well
** Note 2: ** I don’t know if BCT file is backed by RMAN. IF you know , please drop your answer in comments
Note 3: Another great piece of info aboutt BCT I found on some blog The size of the change tracking file is proportional to the size of the database and the number of enabled threads of redo. The size is not related to the frequency of updates to the database.
Typically, the space required for block change tracking is approximately 1/30,000 the size of the data blocks to be tracked. The following two factors that may cause the file to be larger than this estimate suggests: To avoid overhead of allocating space as database grows, the change tracking file size starts at 10MB, and new space is allocated in 10MB increments. Thus, for any database up to approximately 300GB the file size is no smaller than 10MB, for up to approximately 600GB the file size is no smaller than 20MB, and so on. For each datafile, a minimum of 320K of space is allocated in the change tracking file, regardless of the size of the file. Thus, if you have a large number of relatively small datafiles, the change tracking file is larger than for databases with a smaller number of larger datafiles containing the same data.
SELECT FILE#, INCREMENTAL_LEVEL, COMPLETION_TIME, BLOCKS, DATAFILE_BLOCKS FROM V$BACKUP_DATAFILE WHERE INCREMENTAL_LEVEL > 0 AND BLOCKS / DATAFILE_BLOCKS > .5 ORDER BY COMPLETION_TIME;