Oracle Apps DBA

My photo
More than 5+ Years of IT Experience in administering Enterprise wide Multi Vendor UNIX Servers, Oracle Databases (8i to 11gR2), Middle-tiers, Applications and Clusters. I am a Sun Certified System Administrator (SCSA) for Solaris 10, oracle 10g database certified associate(OCA), Oracle 10g Database certified professional (OCP) and Oracle E-Business suite R12 certified professionl(OCP)

Oracle Database and Applications

Sunday, 13 March 2011

Root (/) file system full - ereport.io.ddi.fm-capability

Hi,

Recently we configured LUN from EMC Storage on one of our Apps 11i Staging Instance and ended up with root file system full.

On investigation i found out there are heavy logfiles generating under directory /var/fm/fmd/

Output of df -h command:



bash-3.00# df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c1t2d0s0 9.6G 9.5G 14M 100% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract




File size under /var/fm/fmd directory:



bash-3.00# ls -lrt
total 2220152
drwx------ 2 root sys 512 Jun 20 2010 ckpt
drwx------ 2 root sys 512 Jun 20 2010 rsrc
drwx------ 2 root sys 512 Jun 20 2010 xprt
-rw-r--r-- 1 root root 351 Jun 20 2010 fltlog
-rw-r--r-- 1 root root 10083895 Mar 6 10:52 errlog.4
-rw-r--r-- 1 root root 161641879 Mar 7 20:31 errlog.3
-rw-r--r-- 1 root root 141316343 Mar 9 03:09 errlog.2
-rw-r--r-- 1 root root 683732615 Mar 10 03:10 errlog.1
-rw-r--r-- 1 root root 136814875 Mar 10 06:53 errlog.0
-rw-r--r-- 1 root root 2486803 Mar 14 08:07 errlog
bash-3.00# du -ksh .
1.1G .
bash-3.00#




You couldn't check the content of error log-file as this a binary file. We need to use fmdump utility in order to see the content of this file.



bash-3.00# fmdump -e
Mar 14 08:08:39.2212 ereport.io.ddi.fm-capability 0x69a953fdb5c00401
Mar 14 08:08:39.2212 ereport.io.ddi.fm-capability 0x69a953ff71800801
Mar 14 08:08:39.2221 ereport.io.ddi.fm-capability 0x69a954d64fc00401
Mar 14 08:08:39.2222 ereport.io.ddi.fm-capability 0x69a954f7db400401
Mar 14 08:08:39.2223 ereport.io.ddi.fm-capability 0x69a9550e70800401
Mar 14 08:08:39.2247 ereport.io.ddi.fm-capability 0x69a957511a000401
Mar 14 08:08:39.2247 ereport.io.ddi.fm-capability 0x69a95752eec00801
Mar 14 08:08:40.4236 ereport.io.ddi.fm-capability 0x69adceb085000801
Mar 14 08:08:40.4236 ereport.io.ddi.fm-capability 0x69adceb191c00401
Mar 14 08:08:40.4237 ereport.io.ddi.fm-capability 0x69adcec571400001
Mar 14 08:08:40.4245 ereport.io.ddi.fm-capability 0x69adcf9b62000401
Mar 14 08:08:40.4246 ereport.io.ddi.fm-capability 0x69adcfb3d2400401
Mar 14 08:08:40.4247 ereport.io.ddi.fm-capability 0x69adcfcb80c00401
Mar 14 08:08:43.4370 ereport.io.ddi.fm-capability 0x69b908930bc00801







bash-3.00# fmdump -eV | more
TIME CLASS
Mar 14 2011 07:56:05.511570300 ereport.io.ddi.fm-capability
nvlist version: 0
class = ereport.io.ddi.fm-capability
ena = 0x5eb167dc00400401
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev
device-path = /
(end detector)

dvr-name = fp
__ttl = 0x1
__tod = 0x4d7d9fe5 0x1e7df17c

Mar 14 2011 07:56:05.511570350 ereport.io.ddi.fm-capability
nvlist version: 0
class = ereport.io.ddi.fm-capability
ena = 0x5eb167dc64400801
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev
device-path = /
(end detector)

dvr-name = fp
__ttl = 0x1
__tod = 0x4d7d9fe5 0x1e7df1ae

Mar 14 2011 07:56:06.048252100 ereport.io.ddi.fm-capability
nvlist version: 0
class = ereport.io.ddi.fm-capability
ena = 0x5eb367b0e2000001
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev




After lot of Investigation i found this is a known bug on solaris. As Sun Support is under Oracle umbrella and they have migrated sunsolve.com to myoraclesupport, still SR service for sun not working.


We have to Install patch 141874-09 to overcome this issue. This patch is a core kernel patch so need to restart mahcine and perform a full root FS backup.



bash-3.00# uname -a
SunOS testdb 5.10 Generic_141444-09 sun4u sparc SUNW,Sun-Fire-V440


bash-3.00# patchadd -p | grep 141874-09
bash-3.00#





Installing this patch:



bash-3.00# patchadd 141874-09
Validating patches...

Loading patches installed on the system...

Done!

Loading patches requested to install.

Done!

Checking patches that you specified for installation.

Done!


Approved patches will be installed in this order:

141874-09


Checking installed patches...
Executing prepatch script...
Disabling kernel module unloading ...

Verifying sufficient filesystem capacity (dry run method)...
Installing patch packages...

Patch 141874-09 has been successfully installed.
See /var/sadm/patch/141874-09/log for details
Executing postpatch script...

Patch packages installed:
SUNWcfcl
SUNWcfclr
SUNWfchba
SUNWfcip
SUNWfcmdb
SUNWfcp
SUNWfcsm
SUNWfctl

bash-3.00#




After installation of patch restart the server with reboot -- -r option.

This should solve the issue and shouldn't increase logs under /var/fm/fmd.

Happy Reading :)

1 comment:

Blogger said...

Searching for the Ultimate Dating Site? Create an account and find your perfect match.