Oracle Apps DBA

My photo
More than 5+ Years of IT Experience in administering Enterprise wide Multi Vendor UNIX Servers, Oracle Databases (8i to 11gR2), Middle-tiers, Applications and Clusters. I am a Sun Certified System Administrator (SCSA) for Solaris 10, oracle 10g database certified associate(OCA), Oracle 10g Database certified professional (OCP) and Oracle E-Business suite R12 certified professionl(OCP)

Oracle Database and Applications

Monday, 29 December 2014

R12 : Concurrent Reports in Running Normal Forever

I received an Incident from one our customer that all standard concurrent requests in EBS R12.1.3  are not completing and its continuously in status "Running Normal" forever. This issue was reported on of their recently cloned TEST/UAT Environment.

Environment details:

EBS : 12.1.3
DB   : 11.2.0.1
OS   : IBM-AIX -6.1 (64-bit)

If we check the log file for any standard concurrent request there is no error reported. Below is the concurrent manager logfile for Active users:

Cocurrent log: 
-------------------- 

+---------------------------------------------------------------------------+ 
Application Object Library: Version : 12.0.0 

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved. 

FNDSCURS module: Active Users 
+---------------------------------------------------------------------------+ 

Current system time is 30-OCT-2014 16:25:07 

+---------------------------------------------------------------------------+ 


+----------------------------- 
| Starting concurrent program execution... 
+----------------------------- 


Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are : 
American_America.AR8MSWIN1256 

'.,' 

The concurrent request will stuck  with no further progress.

As a workaround tried restarting database, concurrent manager, restarted all Application services and re linking libraries. But none of them worked.

Cause:

To troubleshoot further tried executing Standard Concurrent request "Active Users" from CLI.

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/bin]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
REP-50125: Caught exception: java.lang.reflect.InvocationTargetException

The problem is with library files "librwu.a or .so"

Tried compiling libraries but failed:

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#make -f ins_reports.mk install
        /oradev/R12_app/apps/tech_st/10.1.2/bin/mkexp.sh /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  > librwuso.exp
        rm -f rwsutil.o rwspid.o ;  /bin/ar x /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  rwsutil.o rwspid.o ;  (LD_OPTIONS="-zmultidefs";  ld -o /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so -bM:SRE -bnoentry -bnolibpath  -bE:librwuso.exp rwsutil.o rwspid.o  -lm -lc -lm  -ldl /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntsh.a -lpthreads  -L/usr/lpp/X11/lib/R6 -L/usr/lpp/X11/lib/R6/Motif2.1 -lXm -lXt -lX11 -lm `cat /oradev/R12_app/apps/tech_st/10.1.2/lib32/sysliblist` /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntst10.a )
ld: 0711-851 SEVERE ERROR: Output file: /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so
        The file is in use and cannot be overwritten.
make: The error code from the last command is 12.


Stop.

To solve this Issue you need to comile libariries with root account as per the MOS tech note "What are the steps needed to Clean up the Libraries during Relinking of Reports on AIX? [ID 834577.1]"

To Implement this workaround i do not have "root" access at that time, so as a simple workaround i have copied the library files from another working TEST enironment and it worked.

After copying these files all standard reports are working as expected.

- Rename existing library files and copy the files from working environment

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2]#cd lib32
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.a librwu.a.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.so librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#cp /usr/tmp/librwu.* .
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Oct 30 23:25 librwu.a
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a.orig
-rwxr-xr-x    1 oradev   oinstall       9758 Oct 30 23:25 librwu.so
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#

- Run concurrent request form command line and test

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
[oradev@TESTBKP[/oradev]#ls

[oradev@TESTBKP[/oradev]#ls -lrt FNDSCURS.txt
-rw-r--r--    1 oradev   oinstall     391895 Oct 30 23:27 FNDSCURS.txt
[oradev@TESTBKP[/oradev]#


Conclusion:

This Issue is reported on IBM-AIX system but it may be applicable on other operating systems as well. I believe library files was corrupted while copying from the source system. 

thanks for reading.

regards,
X A H E E R 

1 comment:

Sridevi K said...

Regards
Sridevi Koduru (Senior Oracle Apps Trainer Oracleappstechnical.com)
LinkedIn profile - https://in.linkedin.com/in/sridevi-koduru-9b876a8b
Please Contact for One to One Online Training on Oracle Apps Technical, Financials, SCM, SQL, PL/SQL, D2K at training@oracleappstechnical.com | +91 - 9581017828.