I received an Incident from one our customer that all standard concurrent requests in EBS R12.1.3 are not completing and its continuously in status "Running Normal" forever. This issue was reported on of their recently cloned TEST/UAT Environment.
Environment details:
EBS : 12.1.3
DB : 12.1.0.2
OS : IBM-AIX -6.1 (64-bit)
If we check the log file for any standard concurrent request there is no error reported. Below is the concurrent manager logfile for Active users:
Cocurrent log: -------------------- +---------------------------------------------------------------------------+ Application Object Library: Version : 12.0.0 Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved. FNDSCURS module: Active Users +---------------------------------------------------------------------------+ Current system time is 30-OCT-2021 16:25:07 +---------------------------------------------------------------------------+ +----------------------------- | Starting concurrent program execution... +----------------------------- Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are : American_America.AR8MSWIN1256 '.,'
The concurrent request will stuck with no further progress.
As a workaround tried restarting database, concurrent manager, restarted all Application services and re linking libraries. But none of them worked.
Cause:
To troubleshoot further tried executing Standard Concurrent request "Active Users" from CLI.
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/bin]#cd [oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt Environment:Environment java.lang.UnsatisfiedLinkError: Can't find library rwu (librwu.a or .so) in sun.boot.library.path or java.library.path sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib Environment:getEnvironmentStrings java.lang.UnsatisfiedLinkError: Can't find library rwu (librwu.a or .so) in sun.boot.library.path or java.library.path sun.boot.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin java.library.path=/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/cz/12.0.0/bin:/oradev/R12_app/apps/tech_st/10.1.2/lib32:/oradev/R12_app/apps/tech_st/10.1.2/lib:/usr/dt/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/classic:/oradev/R12_app/apps/apps_st/appl/sht/12.0.0/lib:/oradev/R12_app/apps/tech_st/10.1.2/jdk/jre/bin/sovvm:/usr/lib REP-50125: Caught exception: java.lang.reflect.InvocationTargetException
The problem is with library files "librwu.a or .so"
Tried compiling libraries but failed:
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#make -f ins_reports.mk install /oradev/R12_app/apps/tech_st/10.1.2/bin/mkexp.sh /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a > librwuso.exp rm -f rwsutil.o rwspid.o ; /bin/ar x /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.a rwsutil.o rwspid.o ; (LD_OPTIONS="-zmultidefs"; ld -o /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so -bM:SRE -bnoentry -bnolibpath -bE:librwuso.exp rwsutil.o rwspid.o -lm -lc -lm -ldl /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntsh.a -lpthreads -L/usr/lpp/X11/lib/R6 -L/usr/lpp/X11/lib/R6/Motif2.1 -lXm -lXt -lX11 -lm `cat /oradev/R12_app/apps/tech_st/10.1.2/lib32/sysliblist` /oradev/R12_app/apps/tech_st/10.1.2/lib32/libclntst10.a ) ld: 0711-851 SEVERE ERROR: Output file: /oradev/R12_app/apps/tech_st/10.1.2/lib32//librwu.so The file is in use and cannot be overwritten. make: The error code from the last command is 12. Stop.
To solve this Issue you need to comile libariries with root account as per the MOS tech note "What are the steps needed to Clean up the Libraries during Relinking of Reports on AIX? [ID 834577.1]"
To Implement this workaround i do not have "root" access at that time, so as a simple workaround i have copied the library files from another working TEST enironment and it worked.
After copying these files all standard reports are working as expected.
- Rename existing library files and copy the files from working environment
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2]#cd lib32 [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.* -rwxr-xr-x 1 oradev oinstall 8868 Mar 7 2008 librwu.a -rwxr-xr-x 1 oradev oinstall 9734 Dec 22 2013 librwu.so [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.a librwu.a.orig [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#mv librwu.so librwu.so.orig [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#cp /usr/tmp/librwu.* . [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#ls -l librwu.* -rwxr-xr-x 1 oradev oinstall 8868 Oct 30 23:25 librwu.a -rwxr-xr-x 1 oradev oinstall 8868 Mar 7 2008 librwu.a.orig -rwxr-xr-x 1 oradev oinstall 9758 Oct 30 23:25 librwu.so -rwxr-xr-x 1 oradev oinstall 9734 Dec 22 2013 librwu.so.orig [oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/lib32]#
- Run concurrent request form command line and test
[oradev@TESTBKP[/oradev/R12_app/apps/tech_st/10.1.2/reports/lib32]#cd [oradev@TESTBKP[/oradev]#$APPLORC userid=apps/appstest mode=character report=$FND_TOP/reports/US/FNDSCURS.rdf batch=yes destype=file desname=FNDSCURS.txt desformat=$FND_TOP/reports/HPL.prt [oradev@TESTBKP[/oradev]#ls [oradev@TESTBKP[/oradev]#ls -lrt FNDSCURS.txt -rw-r--r-- 1 oradev oinstall 391895 Oct 30 23:27 FNDSCURS.txt [oradev@TESTBKP[/oradev]#
Conclusion:
This Issue is reported on IBM-AIX system but it may be applicable on other operating systems as well. I believe library files was corrupted while copying from the source system.
thanks for reading.
regards,
Zaheer
Comments
To create this service, certified nutritionists, personal trainers, and top chefs joined together to develop keto meal plans that are powerful, suitable, cost-efficient, and enjoyable.
From their launch in January 2019, 100's of clients have already remodeled their body and health with the benefits a smart keto diet can give.
Speaking of benefits: clicking this link, you'll discover 8 scientifically-proven ones given by the keto diet.