Issues that we face
Ideally in a production Oracle database, we enable the archivelog mode for the ONLINE redo logs to be archived during log switches. We must also periodically backup and clean up the designated archivelog destination to make room for new archivelogs and avoiding database to go in a hang state.
We normally schedule periodic jobs for these backup and clean up operations. However, there are times when unexpected data load happens in the database without the knowledge of DBA and the application faces this so called database hangs and finally they guys managing the applications knock at the DBA’s door for help. The next course of actions by the DBA is well known to you and I am not going to waste your time in telling all those stories.
What I would like to share today is a very simple method to tackle those unexpected load scenarios where the archivelog destination gets filled up and your periodic backup/cleanup job is yet to be executed.
Oracle provides a very simple and straight forward way for the DBA to configure a ALTERNATE (secondary) location for a specific archive destination in a database. What it means is, if Oracle is not able to archive the ONLINE redo logs to a particular archivelog destination (for any reason be it disk being filled up, disk failure, etcetera) it can route the ONLINE redo logs to be archived in a different location provided it was configured to do so.
How to configure an alternate archive location
Oracle provides a optional attribute with name ALTERNATE that can be used for a particular LOG_ARCHIVE_DEST_n parameter setting. When specified, it enables Oracle to archive the ONLINE redo logs to the alternate location in the event of a failure of the respective primary archivelog location.
There are two steps involved for configuring an alternate archivelog destination.
- We need to first decide which LOG_ARCHIVE_DEST_n destination, we would be using as an alternate destination. Once decided we need to set the state of that destination as alternate as shown below
- Once the alternate destination is determined and its state is changed as ALTERNATE, we can alter our primary archivelog destination (LOG_ARCHIVE_DEST_n) settings to include the alternate destination.
ALTER SYSTEM SET LOG_ARCHIVE_DEST_n='LOCATION=[OS_path_to_alternate_location]' scope=BOTH; ALTER SYSTEM SET LOG_ARCHIVE_DEST_state_n=ALTERNATE scope=BOTH;
Syntax for specifying an alternate location for a primary archivelog destination
ALTER SYSTEM SET LOG_ARCHIVE_DEST_n='LOCATION=path_name' [NOREOPEN/REOPEN=[seconds]] [ MAX_FAILURE=count ] ALTERNATE=LOG_ARCHIVE_DEST_n scope=BOTH;
Few important things to note here, when specifying the alternate archive destination
- When NOREOPEN is specified, Oracle will immediately start archiving the ONLINE redo logs to the alternate location in the event of primary destination failure.
- When REOPEN is specified, Oracle will reattempt to archive the online redo log in primary destination after the number of seconds mentioned in the REOPEN attribute and number of times mentioned in the MAX_FAILURE attribute. Once all the reattempt fails with primary destination, the alternate destination will get activated and Oracle will start archiving the online redo logs to that location. The default values for REOPEN and MAX_FAILURE are 300 seconds and UNLIMITED respectively.
If we want Oracle to immediately failover to the alternate archivelog location, it is recommended to set the NOREOPEN attribute or set REOPEN attribute with a value of 0 (zero), which is equivalent to NOREOPEN.
Lets simulate
Lets validate what we have seen so far with a quick simulation. For instance, I have the following archivelog destination setup in my database
SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ------------------------------ ------- ------------------------------ ------------ --------- LOG_ARCHIVE_DEST_1 PRIMARY /data/archive_primary ALL_ROLES VALID
As of now, I have just one primary archivelog destination configured with LOG_ARCHIVE_DEST_1 parameter and is mapped to the OS location ‘/data/archive_primary’
Solely for the intention of simulation, I have configured the primary archivelog destination against a fully occupied file system.
SQL> !df -h /data/archive_primary Filesystem Size Used Avail Use% Mounted on /dev/sdc1 20G 19G 198M 99% /data
Now, lets consider we would use LOG_ARCHIVE_DEST_2 as the ALTERNATE archivelog destination for our primary archivelog destination LOG_ARCHIVE_DEST_1. First, I need to allocate a OS location to the alternate archivelog destination LOG_ARCHIVE_DEST_2 as shown below.
SQL> alter system set log_archive_dest_2='LOCATION=/bigdata/archive_alternate' scope=both; System altered.
I have used OS location /bigdata/archive_alternate for alternate archivelog destination LOG_ARCHIVE_DEST_2, which has plenty of space available in it.
SQL> !df -h /bigdata/archive_alternate Filesystem Size Used Avail Use% Mounted on /dev/sdf1 20G 46M 19G 1% /bigdata
Now, we need to mark the LOG_ARCHIVE_DEST_2 as alternate archivelog destination as shown below.
SQL> alter system set log_archive_dest_state_2=ALTERNATE scope=both; System altered.
Once, we enable the LOG_ARCHIVE_DEST_2 as alternate archivelog destination, we can verify the status using v$archive_dest as shown below
SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ------------------------------ ------- ------------------------------ ------------ --------- LOG_ARCHIVE_DEST_1 PRIMARY /data/archive_primary ALL_ROLES VALID LOG_ARCHIVE_DEST_2 PRIMARY /bigdata/archive_alternate ALL_ROLES ALTERNATE
At this point, our alternate archivelog destination is ready. However, it is not yet mapped to our primary archivelog destination LOG_ARCHIVE_DEST_1. We can confirm this by querying v$archive_dest as shown below
SQL> select ALTERNATE from v$archive_dest where DEST_NAME='LOG_ARCHIVE_DEST_1'; ALTERNATE ------------------------------ NONE
As per the output, primary archivelog destination LOG_ARCHIVE_DEST_1 doesn’t have a alternate archivelog destination mapped to it yet. In this case, even if we have an alternate archivelog destination configured in our database, it would not be used when the primary archivelog destination fails.
To map the alternate archivelog destination with the primary archivelog destination, we need to alter the settings of our primary archivelog destination as shown below.
SQL> alter system set log_archive_dest_1='LOCATION=/data/archive_primary NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_2' scope=both; System altered.
Now, the alternate archivelog destination LOG_ARCHIVE_DEST_2 is mapped to our primary archivelog destination LOG_ARCHIVE_DEST_1. We can confirm the same by querying v$archive_dest as show below
SQL> select ALTERNATE from v$archive_dest where DEST_NAME='LOG_ARCHIVE_DEST_1'; ALTERNATE ------------------------------ LOG_ARCHIVE_DEST_2
Our database is now ready to tackle the primary archivelog destination failure. I have done few redo extensive queries against the database to induce an artificial failure and here is the log snippet from that failure.
Sun Jul 26 22:21:35 2015 ARC2: Encountered disk I/O error 19502 ARC2: Closing local archive destination LOG_ARCHIVE_DEST_1: '/data/archive_primary/1_361_878342726.dbf' (error 19502) (orlbcp01) Errors in file /app/oracle/diag/rdbms/orlbcp01/orlbcp01/trace/orlbcp01_arc2_4267.trc: ORA-19502: write error on file "/data/archive_primary/1_361_878342726.dbf", block number 2049 (block size=512) ORA-27072: File I/O error ORACLE Instance orlbcp01 - Archival Error ORA-16038: log 1 sequence# 361 cannot be archived ORA-19502: write error on file "", block number (block size=) ORA-00312: online log 1 thread 1: '/data/orlbcp01/redo01.log' Errors in file /app/oracle/diag/rdbms/orlbcp01/orlbcp01/trace/orlbcp01_arc2_4267.trc: ORA-16038: log 1 sequence# 361 cannot be archived ORA-19502: write error on file "", block number (block size=) ORA-00312: online log 1 thread 1: '/data/orlbcp01/redo01.log' Sun Jul 26 22:21:57 2015 Archived Log entry 133 added for thread 1 sequence 361 ID 0xc9377606 dest 2: Archiver process freed from errors. No longer stopped Sun Jul 26 22:22:00 2015 Archived Log entry 134 added for thread 1 sequence 362 ID 0xc9377606 dest 2:
We can observe, upon failure the primary archivelog destination was closed by the Archiver process (ARCH2)
ARC2: Closing local archive destination LOG_ARCHIVE_DEST_1: '/data/archive_primary/1_361_878342726.dbf' (error 19502) (orlbcp01) ORACLE Instance orlbcp01 - Archival Error
Later, the alternate artivelog destination (dest 2) came in to rescue and the archiver was freed from error and log archiving continued on dest 2 (LOG_ARCHIVE_DEST_2).
Archived Log entry 133 added for thread 1 sequence 361 ID 0xc9377606 dest 2: Archiver process freed from errors. No longer stopped
Note: The actual error message in alertlog may vary depending on Oracle database version. This output is from a Oracle 11.2.0.1 database.
Once the alternate archivelog destination is activate due to a failure on the primary destination, the primary destination would be marked as DISABLED and the alternate destination will become the VALID archivelog destination. We can confirm the same by querying v$archive_dest as follows.
SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ------------------------------ ------- ------------------------------ ------------ --------- LOG_ARCHIVE_DEST_1 PRIMARY /data/archive_primary ALL_ROLES DISABLED LOG_ARCHIVE_DEST_2 PRIMARY /bigdata/archive_alternate ALL_ROLES VALID
At this point the database archivelogs are going to the alternate destination (LOG_ARCHIVE_DEST_2) and it will continue to go there unless it is not instructed to do otherwise. Even if we free up the space in primary archivelog destination, Oracle will not route the archivelogs automatically back to the primary destination.
Once the space is freed on the primary archivelog destination, we need to manually revert back to the original archivelog setup as follows
------- ------- Enable the primary archivelog destination (was disabled due to failure) SQL> alter system set log_archive_dest_state_1=ENABLE scope=both; System altered. ------ ------ Mark the designated alternate destination as ALTERNATE (became primary due to failure of actual primary destination ) SQL> alter system set log_archive_dest_state_2=ALTERNATE scope=both; System altered.
We can confirm the settings by querying v$archive_dest as follows
SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ------------------------------ ------- ------------------------------ ------------ --------- LOG_ARCHIVE_DEST_1 PRIMARY /data/archive_primary ALL_ROLES VALID LOG_ARCHIVE_DEST_2 PRIMARY /bigdata/archive_alternate ALL_ROLES ALTERNATE SQL> select ALTERNATE from v$archive_dest where DEST_NAME='LOG_ARCHIVE_DEST_1'; ALTERNATE ------------------------------ LOG_ARCHIVE_DEST_2 SQL>
We are now back in our original archivelog settings. Oracle will now route the archivelogs to our primary archivelog destination. We can confirm this from the alertlog as show below
Sun Jul 26 23:00:16 2015 Archived Log entry 138 added for thread 1 sequence 366 ID 0xc9377606 dest 1:
As we can observe, archivelogs are now going to primary destination dest 1 (LOG_ARCHIVE_DEST_1).
Conclusion
Never settle down with just the primary archivelog destination. Oracle has given us the option to have alternatives. Why not use it? It can save the database from an outage!!