R12 : Concurrent Reports in Running Normal Forever

-
I received an Incident from one our customer that all standard concurrent requests in EBS R12.1.3  are not completing and its continuously in status "Running Normal" forever. This issue was reported on of their recently cloned TEST/UAT Environment.

Environment details:

EBS : 12.1.3
DB   : 12.1.0.2
OS   : IBM-AIX -6.1 (64-bit)

If we check the log file for any standard concurrent request there is no error reported. Below is the concurrent manager logfile for Active users:

Cocurrent log: 
-------------------- 

+---------------------------------------------------------------------------+ 
Application Object Library: Version : 12.0.0 

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved. 

FNDSCURS module: Active Users 
+---------------------------------------------------------------------------+ 

Current system time is 30-OCT-2021 16:25:07 

+---------------------------------------------------------------------------+ 


+----------------------------- 
| Starting concurrent program execution... 
+----------------------------- 


Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are : 
American_America.AR8MSWIN1256 

'.,' 

The concurrent request will stuck  with no further progress.

As a workaround tried restarting database, concurrent manager, restarted all Application services and re linking libraries. But none of them worked.

Cause:

To troubleshoot further tried executing Standard Concurrent request "Active Users" from CLI.

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/bin]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu  (librwu.a or .so) in sun.boot.library.path or java.library.path
sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin
java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib
REP-50125: Caught exception: java.lang.reflect.InvocationTargetException

The problem is with library files "librwu.a or .so"

Tried compiling libraries but failed:

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#make -f ins_reports.mk install
        /oradev/R12_app/apps/tech_st/10.1.2/bin/mkexp.sh /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  > librwuso.exp
        rm -f rwsutil.o rwspid.o ;  /bin/ar x /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a  rwsutil.o rwspid.o ;  (LD_OPTIONS="-zmultidefs";  ld -o /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so -bM:SRE -bnoentry -bnolibpath  -bE:librwuso.exp rwsutil.o rwspid.o  -lm -lc -lm  -ldl /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntsh.a -lpthreads  -L/usr/lpp/X11/lib/R6 -L/usr/lpp/X11/lib/R6/Motif2.1 -lXm -lXt -lX11 -lm `cat /oradev/R12_app/apps/tech_st/10.1.2/lib32/sysliblist` /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntst10.a )
ld: 0711-851 SEVERE ERROR: Output file: /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so
        The file is in use and cannot be overwritten.
make: The error code from the last command is 12.


Stop.

To solve this Issue you need to comile libariries with root account as per the MOS tech note "What are the steps needed to Clean up the Libraries during Relinking of Reports on AIX? [ID 834577.1]"

To Implement this workaround i do not have "root" access at that time, so as a simple workaround i have copied the library files from another working TEST enironment and it worked.

After copying these files all standard reports are working as expected.

- Rename existing library files and copy the files from working environment

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2]#cd lib32
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.a librwu.a.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.so librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#cp /usr/tmp/librwu.* .
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.*
-rwxr-xr-x    1 oradev   oinstall       8868 Oct 30 23:25 librwu.a
-rwxr-xr-x    1 oradev   oinstall       8868 Mar  7 2008  librwu.a.orig
-rwxr-xr-x    1 oradev   oinstall       9758 Oct 30 23:25 librwu.so
-rwxr-xr-x    1 oradev   oinstall       9734 Dec 22 2013  librwu.so.orig
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#

- Run concurrent request form command line and test

[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#cd
[oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt
[oradev@TESTBKP[/oradev]#ls

[oradev@TESTBKP[/oradev]#ls -lrt FNDSCURS.txt
-rw-r--r--    1 oradev   oinstall     391895 Oct 30 23:27 FNDSCURS.txt
[oradev@TESTBKP[/oradev]#

Conclusion:

This Issue is reported on IBM-AIX system but it may be applicable on other operating systems as well. I believe library files was corrupted while copying from the source system. 

thanks for reading.

regards,
Zaheer

Comments

Blogger said…
If you're trying to burn fat then you certainly need to get on this totally brand new personalized keto diet.

To create this service, certified nutritionists, personal trainers, and top chefs joined together to develop keto meal plans that are powerful, suitable, cost-efficient, and enjoyable.

From their launch in January 2019, 100's of clients have already remodeled their body and health with the benefits a smart keto diet can give.

Speaking of benefits: clicking this link, you'll discover 8 scientifically-proven ones given by the keto diet.
23 February 2020 at 19:06
Post a Comment

Popular posts from this blog

Disable Firewall on Oracle Linux 8

-
Image
In this blogpost we will see how we can stop/disable the firewall on Oracle Linux 8, the firewall command is same in both linux 7 an linux 8. The below listed is the procedure for stopping and disabling the  firewall on Oracle Linux 8. - Here we can see the firewall deamon in active state - Here when we stop the firewall in previous command, now the firewall daemon is dead - For permanent disabling the firewall on server, we can use "disable" option The following commands will be helpful: #systemctl status firewalld #systemctl stop firewalld #systemctl disable firewalld #systemctl enable firewalld #systemctl start firewalld Hope it helps !! Thanks for reading :) regards, X A H E E R
Read more

Installation of Oracle Applications R12.1.1 on Linux and vmware

-
Image
The below article is step by step procedure to Install oracle E-Business suite R12.1.1 on Oracle Enterprise Linux 5.6 (32-bit) using vmware server 2 on windows 7 64 bit. This is a single node Installation. (All services will be hosting on same machine 0 db, conc, web, forms etc) I have Installed it on a virtual machine with 4gb ram, 120 GB of local disk space and 1 core cpu. - Requires 80 GB of free space for fresh installation (excluding stage area) - Requires 100+ GB of free space for vision installation (excl. stage) I will split this configuration in 5 steps: 1) create and configure virtual machine 2) Install linux operating system 3) create stage Area 4) Perform all operating system per-requisites 5) Install oracle E-business suite R12.1.1 create and configure virtual machine: >> Login to vmware Server console >> click on add virtual machine >> Here i have selected iso image for OS Installation, you can select local cd drive if u want. >> Now the virtual m...
Read more

Enable Desktop on Oracle Solaris 11.4

-
Image
Oracle Solaris 11 installation has multiple options to choose for installation of an Operating Environment, but mostly Oracle Solaris text install media is used and this installation media doesn't offer the GUI Desktop Environment by default after the installation. This blog post will explain how we can enable the desktop for Oracle Solaris 11.4 operating system, after the completion of installation. We have to install "solaris-desktop" package and reboot the machine and GUI desktop will be enabled for the Operating System. In this blog post my virtual machine is connected to the internet and hence I am able to use available pupblic repository for package installation, if in case internet is not available for the server/machine then we have configure the local/Server  repository for the installation. Follow the below steps for desktop package installation: After installation of dekstop package we are now able to login with GUI desktop environment...
Read more