Skip to main content

R12 : Concurrent Reports in Running Normal Forever

I received an Incident from one our customer that all standard concurrent requests in EBS R12.1.3  are not completing and its continuously in status "Running Normal" forever. This issue was reported on of their recently cloned TEST/UAT Environment.

Environment details:

EBS : 12.1.3
DB   : 12.1.0.2
OS   : IBM-AIX -6.1 (64-bit)

If we check the log file for any standard concurrent request there is no error reported. Below is the concurrent manager logfile for Active users:

Cocurrent log: 
-------------------- 

+---------------------------------------------------------------------------+ 
Application Object Library: Version : 12.0.0 

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved. 

FNDSCURS module: Active Users 
+---------------------------------------------------------------------------+ 

Current system time is 30-OCT-2021 16:25:07 

+---------------------------------------------------------------------------+ 


+----------------------------- 
| Starting concurrent program execution... 
+----------------------------- 


Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are : 
American_America.AR8MSWIN1256 

'.,' 

The concurrent request will stuck  with no further progress.

As a workaround tried restarting database, concurrent manager, restarted all Application services and re linking libraries. But none of them worked.

Cause:

To troubleshoot further tried executing Standard Concurrent request "Active Users" from CLI.

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/bin]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
REP-50125: Caught exception: java.lang.reflect.InvocationTargetException

The problem is with library files "librwu.a or .so"

Tried compiling libraries but failed:

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#make -f ins_reports.mk install
        /oradev/R12_app/apps/tech_st/10.1.2/bin/mkexp.sh /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  > librwuso.exp
        rm -f rwsutil.o rwspid.o ;  /bin/ar x /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  rwsutil.o rwspid.o ;  (LD_OPTIONS="-zmultidefs";  ld -o /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so -bM:SRE -bnoentry -bnolibpath  -bE:librwuso.exp rwsutil.o rwspid.o  -lm -lc -lm  -ldl /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntsh.a -lpthreads  -L/usr/lpp/X11/lib/R6 -L/usr/lpp/X11/lib/R6/Motif2.1 -lXm -lXt -lX11 -lm `cat /oradev/R12_app/apps/tech_st/10.1.2/lib32/sysliblist` /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntst10.a )
ld: 0711-851 SEVERE ERROR: Output file: /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so
        The file is in use and cannot be overwritten.
make: The error code from the last command is 12.


Stop.

To solve this Issue you need to comile libariries with root account as per the MOS tech note "What are the steps needed to Clean up the Libraries during Relinking of Reports on AIX? [ID 834577.1]"

To Implement this workaround i do not have "root" access at that time, so as a simple workaround i have copied the library files from another working TEST enironment and it worked.

After copying these files all standard reports are working as expected.

- Rename existing library files and copy the files from working environment

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2]#cd lib32
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.a librwu.a.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.so librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#cp /usr/tmp/librwu.* .
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Oct 30 23:25 librwu.a
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a.orig
-rwxr-xr-x    1 oradev   oinstall       9758 Oct 30 23:25 librwu.so
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#

- Run concurrent request form command line and test

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
[oradev@TESTBKP[/oradev]#ls

[oradev@TESTBKP[/oradev]#ls -lrt FNDSCURS.txt
-rw-r--r--    1 oradev   oinstall     391895 Oct 30 23:27 FNDSCURS.txt
[oradev@TESTBKP[/oradev]#


Conclusion:

This Issue is reported on IBM-AIX system but it may be applicable on other operating systems as well. I believe library files was corrupted while copying from the source system. 

thanks for reading.

regards,
Zaheer

Comments

Blogger said…
If you're trying to burn fat then you certainly need to get on this totally brand new personalized keto diet.

To create this service, certified nutritionists, personal trainers, and top chefs joined together to develop keto meal plans that are powerful, suitable, cost-efficient, and enjoyable.

From their launch in January 2019, 100's of clients have already remodeled their body and health with the benefits a smart keto diet can give.

Speaking of benefits: clicking this link, you'll discover 8 scientifically-proven ones given by the keto diet.

Popular posts from this blog

Disable Firewall on Oracle Linux 8

In this blogpost we will see how we can stop/disable the firewall on Oracle Linux 8, the firewall command is same in both linux 7 an linux 8. The below listed is the procedure for stopping and disabling the  firewall on Oracle Linux 8. - Here we can see the firewall deamon in active state - Here when we stop the firewall in previous command, now the firewall daemon is dead - For permanent disabling the firewall on server, we can use "disable" option The following commands will be helpful: #systemctl status firewalld #systemctl stop firewalld #systemctl disable firewalld #systemctl enable firewalld #systemctl start firewalld Hope it helps !! Thanks for reading :) regards, X A H E E R

Enable Desktop on Oracle Solaris 11.4

Oracle Solaris 11 installation has multiple options to choose for installation of an Operating Environment, but mostly Oracle Solaris text install media is used and this installation media doesn't offer the GUI Desktop Environment by default after the installation. This blog post will explain how we can enable the desktop for Oracle Solaris 11.4 operating system, after the completion of installation. We have to install "solaris-desktop" package and reboot the machine and GUI desktop will be enabled for the Operating System. In this blog post my virtual machine is connected to the internet and hence I am able to use available pupblic repository for package installation, if in case internet is not available for the server/machine then we have configure the local/Server  repository for the installation. Follow the below steps for desktop package installation: After installation of dekstop package we are now able to login with GUI desktop environment

Oracle Applications R12.2 Installation on Linux using virtual box

Oracle E-Business suite R12.2 is released a month ago. I received too many requests from my followers to demonstrate this Installation. This is a step by step detailed article for Installing Oracle E-business suite R12,2 on Oracle Enterprise Linux 5.7 (64-bit). I've used oracle virtual box for Installing Linux. Note:  This Installation is demonstrated on virtual machine with 8GB RAM and Installation completed in 7 hours. I am not sure whether this Installation will work with less than this .  My Virtual Machine configuration: CPU    = 1 core RAM   = 8 GB HDD   =  380 GB Software's Required: 1) Oracle Virtual Box 2) Oracle Enterprise Linux 5.7 (64-bit) 3) Oracle E-Business suite R12.2 for LinuxX86-64-bit Installation Steps: Hardware requirements Create virtual machine and Install operating system Operating System pre-requisites Prepare Stage Area Install Oracle EBS R12.2 software Verify Installation 1. Hardware Requirement