RMAN BACKUP AS COPY failing with ORA-00600 [kfioTranslateIO03] [17090]

Recently, I migrating a standalone database 'rnwphoto' from file system to ASM. I decided to go with the BACKUP AS COPY strategy, to save on downtime. But I was stuck for a long time trying to make the copy, but was being haunted by its failure.

Let's get to the background information. 

Database Standalone Home: /u01/app/oracle/product/11.2.0/dbhome_1 --> This is the Oracle Home used by rnwphoto database.

To convert the database to ASM based RAC, we installed GI in the GRID_HOME (/u01/app/11.2.0/grid) and RAC DB Home (/u01/app/oracle/product/11.2.0/dbhome_2)

Issue:

RMAN> BACKUP AS COPY DATABASE FORMAT '+DATA_DG'; 

Starting backup at 09-SEP-15 
allocated channel: ORA_DISK_1 
channel ORA_DISK_1: SID=8 device type=DISK 
allocated channel: ORA_DISK_2 
channel ORA_DISK_2: SID=69 device type=DISK 
allocated channel: ORA_DISK_3 
channel ORA_DISK_3: SID=132 device type=DISK 
allocated channel: ORA_DISK_4 
channel ORA_DISK_4: SID=72 device type=DISK 
channel ORA_DISK_1: starting datafile copy 
input datafile file number=00001 name=/u01/app/oracle/rnwphoto/system01.dbf 
channel ORA_DISK_2: starting datafile copy 
input datafile file number=00002 name=/u01/app/oracle/rnwphoto/sysaux01.dbf 
channel ORA_DISK_3: starting datafile copy 
input datafile file number=00005 name=/u01/app/oracle/rnwphoto/example01.dbf 
channel ORA_DISK_4: starting datafile copy 
input datafile file number=00003 name=/u01/app/oracle/rnwphoto/undotbs01.dbf 
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/09/2015 11:12:22 
RMAN-10038: database session for channel ORA_DISK_1 terminated unexpectedly 
channel ORA_DISK_1 disabled, job failed on it will be run on another channel 
RMAN-03009: failure of backup command on ORA_DISK_2 channel at 09/09/2015 11:12:22 
RMAN-10038: database session for channel ORA_DISK_2 terminated unexpectedly 
channel ORA_DISK_2 disabled, job failed on it will be run on another channel 
RMAN-03009: failure of backup command on ORA_DISK_3 channel at 09/09/2015 11:12:22 
RMAN-10038: database session for channel ORA_DISK_3 terminated unexpectedly 
channel ORA_DISK_3 disabled, job failed on it will be run on another channel 
RMAN-00571: =========================================================== 
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== 
RMAN-00571: =========================================================== 
RMAN-03009: failure of backup command on ORA_DISK_4 channel at 09/09/2015 11:12:22 
RMAN-10038: database session for channel ORA_DISK_4 terminated unexpectedly 

ASM alertlog didnt show any errors or warnings.
In the database alertlog you will see that each time the RMAN command fails with  --> ORA-00600 [kfioTranslateIO03] [17090] 

A similar issue and it's resolution is shown in the Doc ID 1336846.1 
As per this document, this issue occurs due to multiple reasons and one of them is a permission issue... 

1. Check the group owner required on 'oracle' binary file to access ASM successfully:

[grid@node1-riz lib]$ cat $ORACLE_HOME/lib/config.c | grep ASM 
#define SS_ASM_GRP "asmadmin"     >>>> 
char *ss_dba_grp[] = {SS_DBA_GRP, SS_OPER_GRP, SS_ASM_GRP}; 

2. Let's check the group owner of all ORACLE_HOME/bin/oracle binary:

Grid Home:
[grid@node1-riz ~]$ ls -latr $GRID_HOME/bin/oracle 
-rwsr-s--x 1 grid oinstall 169802036 Sep 3 12:22 /u01/app/11.2.0/grid/bin/oracle 

RAC Database Home:
[oracle@node1-riz ~]$ ls -lart /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle -rwsr-s--x 1 oracle asmadmin 192296439 Sep 3 13:30 /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle

rnwphoto Standalone Database Home: 
[oracle@node1-riz ~]$ ls -lart $ORACLE_HOME/bin/oracle 
-rwsr-s--x 1 oracle oinstall 192296337 Sep 2 19:41 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle 
>>>>> *****************************************************  <<<<<

Do you see it? If not, read ahead...

Point #1 states that 'asmadmin' should be the group owner of the $ORACLE_HOME/bin/oracle

Let's ignore the Grid Home findings - as that's not a part of the problem. It's just for your information.

RAC Database Home works fine with the ASM -- as the group owner of the 'oracle' binary is 'asmadmin'

Standalone Database Home was created much before there was a Grid Home, so it still has 'oinstall' as the group owner of the 'oracle' binary file. 

Solution:
In the Standalone Oracle Home, change the ownership of the 'oracle' binary from oracle:oinstall to oracle:asmadmin --- and WOOOLA !!! 

WARNING: You need a downtime to make this change. If you change the ownership without shutting the database(s), then expect all the databases running on that ORACLE_HOME to CRASH!!!

My Photography Slideshow

Many of you may or may not know, that I am not only passionate about Oracle and new technologies, but also passionate about Photography. 

I am working on creating a small slideshow to showcase some of my special work - just to keep you even more relaxed. 

If you wish to check it out, you can visit my:

Facebook Page: https://www.facebook.com/rizwanwangdephotography
500px: https://500px.com/rizwanwangde

I'm back...

Hi Folks,

I am back ! This blog has been dormant for way too long now. There are so many things I could have shared in the past year but lost on all those opportunities. I promise to make it up, and post as regularly as possible. 

Other than the daily stuff, my short term major goals are as below:

1. EM12cR3 Setup on HP-UX 

2. DataGuard 12c Setup - ASM <--> Filesystem

Regards,

Rizwan Wangde

Oracle 10g Undo Auto Retention Bug causing continuous increase in UNDO tablespace utilization

Something interesting happened the other day on one of our Production Databases. The UNDO tablespace just kept filling up, without showing any downward trend. As this was a mission critical database, core-banking, we couldn't take the risk.

I check the usage of UNDO and for any other abnormal sessions or jobs that might be out of place. Nothing!!! But the space usage just kept increasing. I realized this had to be some sort of a bug. Well, after so much experience on the Oracle Databases, a DBA must have the intuition that "this must be a bug!". It was weird because the Undo Segments just wouldn't Expire... All were in the Active state.

I checked the Metalink for some time, and found a document which was very closely related to my situation.

Bug 5387030 Automatic tuning of undo_retention causes unusual extra space allocation 

Problem Description:

When undo tablespace is using NON-AUTOEXTEND datafiles,V$UNDOSTAT.TUNED_UNDORETENTION may be calculated too high preventing undo block from being expired and reused. In extreme cases the undo tablespace could be filled to capacity by these unexpired blocks.

To find out if you are hitting the same, check the output of the below statement:

select end_time, MAXQUERYLEN, TUNED_UNDORETENTION 

from v$undostat where

trunc(end_time)>trunc(sysdate-1) order by 1,2,3;

This will show a comparison between the maxquerylen and the tuned_undoretention. Ideally, the maximum amount of time set by the automatic Undo Retention should be either the maxquerylen or the value of undo_retention parameter. But in this case, the values under TUNED_UNDORETENTION column may turn out to be much higher than the both the maxquerylen or the value of undo_retention parameter

Solution:

Set the below hidden parameter:

alter system set "_smu_debug_mode" = 33554432 scope=both;

You can see that all the UNDO segments that were ACTIVE and well passed the UNDO_RETENTION value have finally expired, and the space released from the UNDO Tablespace.

Oracle Database

ORA-01017: invalid username/password when connecting from a 9i ODBC driver to an 11g Database

Hi folks,

HP Openview, used for monitoring and alerting, is in it's migration phase. The database now runs on Oracle 11g. However, the Reporting Server that connects to the database underwent no such migration or upgrade. This Reporting Server is still using the Oracle 9.1 ODBC drivers.

While trying to connect to the new 11g database, the team encountered the following error:

Unable to connect
SQLState=28000
[Oracle][ODBC][Ora]ORA-01017: invalid username/password; logon denied

Interestingly, if I tried to connect using SQLPLUS, I could establish a connection without any problems.


C:\Documents and Settings\xxxxxx>sqlplus

SQL*Plus: Release 9.2.0.1.0 - Production on Wed Sep 5 10:13:57 2012

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Enter user-name: opc_report@xxxxxxxxxxxxx
Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select * from v$version;

BANNER
-------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
PL/SQL Release 11.2.0.3.0 - Production
CORE    11.2.0.3.0      Production
TNS for HPUX: Version 11.2.0.3.0 - Production
NLSRTL Version 11.2.0.3.0 - Production

SQL>

Cause and Solution:

This was because of the usage of CASE SENSITIVE passwords in Oracle 11g. This was resolved by turning off the case sensitive feature in the database:

$ sqlplus

SQL*Plus: Release 11.2.0.3.0 Production on Wed Sep 5 10:05:06 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Enter user-name: / as sysdba

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL> alter system set SEC_CASE_SENSITIVE_LOGON = FALSE
  2  scope=both;

System altered.

And, woolaaa:

Windows 8 and the Blank Laptop LCD Screen issue

I recently installed the leaked Windows 8 RTM, to try it out before I purchase (and I will most definitely buy it as it cost only 15$ for the upgrade).
I have worked on the Windows 8 for 2 days now, and I think "THIS IS IT!". Windows 8 is one of the most amazing and an innovative platform of OS' that Microsoft has ever released. It's blazing FAST, beautiful and easy to use. I know that the first time users will find it a little difficult to get used to the new style of Navigation, but 2 days in and you'll love everything about Windows.

Blank LCD/Screen issue on loading Windows 8:
Many of you might be facing this issue - after Windows 8 completes loading, you will see a blank screen instead of the Login Screen. When I connect the laptop to an external LCD monitor, it works fine. But the Laptop's LCD just wont!

I tried a lot of tricks to fix it, but this is what clicked at the end:

1. Put your laptop into Sleep Mode by pressing the Power Button once or you can just close the lid of your laptop. 

2. Wake your Laptop from the Sleep Mode by pressing the Power Button again or opening the LCD lid. 

Woola... I can see my screen now!

I had to do this 2-3 times, and after that I have rebooted, or hard reset my laptop many times without facing the Blank LCD issue.

Let me know if you faced this issue and if the above trick helped you.