OEM 13c: The Next-Gen OPATCH version 13.9 and issues on Solaris 10 - "opatch: test: unknown operator =="

To those running OEM 13c - you may have noticed the bug "ORA-06553: PLS-306: wrong number or types of arguments in call to 'REPORT_SQL_MONITOR_LIST'" causing the SQL Monitoring page to fail.

Rejoice - the fix is now available "Oracle® Management Service 13c Release 2 (13.2.0.0.0) System Patch Patch for Bug # 25197714 - Enterprise Manager for OMS Plug-ins 13.2.0.0.0"

Readme --> HERE

This post is to do mostly with "Pre-requisite #5":
-----
Ensure that you have the latest version of OPatch 13.9.0.0.0 and OMSPatcher 13.8.0.0.1 on all OMS instance platform homes.
If you are not sure about the OPatch version you have, or to download the latest version, follow the instructions outlined in the My Oracle Support note 224346.1 available at:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1

If you are not sure about the OMSPatcher version you have, or to download the latest version, follow the instructions outlined in the My Oracle Support note 2203007.1 available at:
https://support.oracle.com/epmos/faces/DocumentDisplay?id=2203007.1
----

OMSPatcher upgrade is simple - download --> unzip --> replace folder [the usual (but now historical) procedure used to upgrade OPatch].

Hang on, what do you mean by "historical" procedure? What's new with OPatch upgrade? 

OPatch now comes bundled in a 'jar' file that needs to be installed using at least JRE 1.7. Luckily, OMS 13c installation comes built in with this version of JAVA.

Command is as below:
[oracle->ogcoms]/u01/software/6880880]$OMS_HOME/oracle_common/jdk/bin/java -jar -d64 opatch_generic.jar -silent oracle_home=$OMS_HOME

(-d64 parameter is required if you are installing OPatch in a 64 bit setup)

Should work like a charm for any other OS, except Solaris (as of the version 13.9).

Issue:
Upgrade was successful without any errors. But, "opatch" commands were failing:

[oracle->ogcoms]/u01/app/oracle/middleware13c/OPatch]./opatch 
./opatch: test: unknown operator == 
[oracle->ogcoms]/u01/app/oracle/middleware13c/OPatch]./opatch version 
./opatch: test: unknown operator == 

This is a bug! (yes, yet another bug!)

Bug 16393828 [ - OPATCH.SH FAILS ON SOLARIS
Bug 16393904 [ - OPATCH-NEXTGEN: OPATCH NOT WORKING IN SOLARIS SPARC 10

opatch is an ascii file, so looking for the text "==" in the file gives you the faulty line:

[oracle->ogcoms]/home/oracle/OPatch]grep == opatch
#    ==> yes: ready to 'callOPatch'
#    ==> no: go to #2
#    ==> yes: ready to 'callOpatch'
#    ==> no: go to #3
#    ==> yes: ready to 'callOpatch'
#    ==> no: Error and Exit
   if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ];then  
# JDK 8:  MaxPermSize <=== replaced by ===> MaxMetaspaceSize:  default = 128M
   # if JDK8 and has flag 'MaxPermSize', replace MaxPermSize <==> MaxMetaspaceSize

Faulty line is:

   if [ "$PLATFORM" == "Linux" -a "$ARCH" == "x86_64" ];then

Solution:
Issue with Solaris server is that "==" will not work, therefore I modified it to "=" in the opatch and opatch.pl files:

   if [ "$PLATFORM" = "Linux" -a "$ARCH" = "x86_64" ];then

I tried opatch command again and voola - we have a perfectly working OPatch!

[oracle->ogcoms]/u01/app/oracle/middleware13c/OPatch]./opatch version 
OPatch Version: 13.9.1.0.0 

OPatch succeeded. 

I miss the "non-buggy", simple unzip and replace OPatch directory method!

ORA-600 [ktu_format_nr no disk align] - Oracle 12c Database Migration stuck at Convert Datafile of UNDO tablespace

Bugs Bugs Bugs... 3rd bug this month!

So, I was performing a simple cross-platform Database Migration from "Solaris[tm] OE (64-bit)" to "HP-UX IA (64-bit)". As both the source and the target OS runs on the BIG endian format, the migration using "RMAN transportable database" is the simplest - requiring to convert only the datafiles/tablespaces containing some kind of UNDO segment.

Steps for this migration can be found at:
Cross-Platform Database Migration (across same endian) using RMAN Transportable Database (Doc ID 1401921.1)

Now, I had 1 file from SYSTEM tablespace and a few from UNDOTBS_01 to convert.
Most of them converted without a hickup, but one file "undotbs_01_01.dbf" would get stuck while conversion.

RMAN> CONVERT 
DATAFILE '/ofdmdb/STAGE_DIR/undotbs_01_01.dbf' 
FORMAT '/ofdmdb/undotbs_01_01.dbf' 
FROM PLATFORM 'Solaris[tm] OE (64-bit)' ; 

Starting conversion at target at 18-JAN-17 
using channel ORA_DISK_1 
channel ORA_DISK_1: starting datafile conversion 
input file name=/ofdmdb/STAGE_DIR/undotbs_01_01.dbf 

--- -- - Stuck here since 2 hours and counting... 

I check the alert.log and found the below:

Thu Jan 19 15:06:46 2017 
Errors in file /u01/app/oracle/diag/rdbms/ofdm/ofdm/trace/ofdm_ora_16742.trc (incident=65019): 
----->>> ORA-00600: internal error code, arguments: [ktu_format_nr no disk align], [104], [33281], [45056], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/ofdm/ofdm/incident/incdir_65019/ofdm_ora_16742_i65019.trc 
Thu Jan 19 15:06:49 2017 
Dumping diagnostic data in directory=[cdmp_20170119150649], requested by (instance=1, osid=9513668), summary=[incident=65019]. 
Use ADRCI or Support Workbench to package the incident. 
See Note 411.1 at My Oracle Support for error and packaging details. 
Thu Jan 19 15:07:08 2017 
Sweep [inc][65019]: completed 
Sweep [inc2][65019]: completed 

On searching for related documents I found that this was a Bug and there were no straight patches available yet:

Bug 24332831 - RMAN: ORA-600 [ktu_format_nr no disk align] while convert from HP-UX ( Doc ID 24332831.8 ) 
or a more accessible post:
http://remidian.com/2012/07/transporting-an-oracle-database-to-another-os-platform-the-fastest-way

I then had to raise a ticket with Oracle Support and they prepared a bugfix patch (Patch 24332831 for HP-UX) for us based on our version and patch level.

I then applied the patch using the usual "opatch apply", and then tried to convert the file again:


oracle->sysdev]/u01/software/bugfix/24332831>rman target /
Recovery Manager: Release 12.1.0.2.0 - Production on Mon Jan 23 09:06:45 2017
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.
connected to target database: OFDM (not mounted)

RMAN> CONVERT DATAFILE '/ofdmdb/STAGE_DIR/undotbs_01_01.dbf'
FORMAT '/ofdmdb/undotbs_01_01.dbf'
FROM PLATFORM 'Solaris[tm] OE (64-bit)' ;2> 3>

Starting conversion at target at 23-JAN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=122 device type=DISK
channel ORA_DISK_1: starting datafile conversion
input file name=/ofdmdb/STAGE_DIR/undotbs_01_01.dbf
converted datafile=/ofdmdb/undotbs_01_01.dbf
channel ORA_DISK_1: datafile conversion complete, elapsed time: 00:00:15
Finished conversion at target at 23-JAN-17

Finally, I proceeded with the next steps in migration without another hiccup.