Root (/) file system full - ereport.io.ddi.fm-capability

Hi,

Recently we configured LUN from EMC Storage on one of our Apps 11i Staging Instance and ended up with root file system full.

On investigation i found out there are heavy logfiles generating under directory /var/fm/fmd/

Output of df -h command:

bash-3.00# df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t2d0s0      9.6G   9.5G    14M   100%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract

File size under /var/fm/fmd directory:

bash-3.00# ls -lrt
total 2220152
drwx------   2 root     sys          512 Jun 20  2010 ckpt
drwx------   2 root     sys          512 Jun 20  2010 rsrc
drwx------   2 root     sys          512 Jun 20  2010 xprt
-rw-r--r--   1 root     root         351 Jun 20  2010 fltlog
-rw-r--r--   1 root     root     10083895 Mar  6 10:52 errlog.4
-rw-r--r--   1 root     root     161641879 Mar  7 20:31 errlog.3
-rw-r--r--   1 root     root     141316343 Mar  9 03:09 errlog.2
-rw-r--r--   1 root     root     683732615 Mar 10 03:10 errlog.1
-rw-r--r--   1 root     root     136814875 Mar 10 06:53 errlog.0
-rw-r--r--   1 root     root     2486803 Mar 14 08:07 errlog
bash-3.00# du -ksh .
 1.1G   .
bash-3.00#

You couldn't check the content of error log-file as this a binary file. We need to use fmdump utility in order to see the content of this file.

bash-3.00# fmdump -e
Mar 14 08:08:39.2212 ereport.io.ddi.fm-capability          0x69a953fdb5c00401
Mar 14 08:08:39.2212 ereport.io.ddi.fm-capability          0x69a953ff71800801
Mar 14 08:08:39.2221 ereport.io.ddi.fm-capability          0x69a954d64fc00401
Mar 14 08:08:39.2222 ereport.io.ddi.fm-capability          0x69a954f7db400401
Mar 14 08:08:39.2223 ereport.io.ddi.fm-capability          0x69a9550e70800401
Mar 14 08:08:39.2247 ereport.io.ddi.fm-capability          0x69a957511a000401
Mar 14 08:08:39.2247 ereport.io.ddi.fm-capability          0x69a95752eec00801
Mar 14 08:08:40.4236 ereport.io.ddi.fm-capability          0x69adceb085000801
Mar 14 08:08:40.4236 ereport.io.ddi.fm-capability          0x69adceb191c00401
Mar 14 08:08:40.4237 ereport.io.ddi.fm-capability          0x69adcec571400001
Mar 14 08:08:40.4245 ereport.io.ddi.fm-capability          0x69adcf9b62000401
Mar 14 08:08:40.4246 ereport.io.ddi.fm-capability          0x69adcfb3d2400401
Mar 14 08:08:40.4247 ereport.io.ddi.fm-capability          0x69adcfcb80c00401
Mar 14 08:08:43.4370 ereport.io.ddi.fm-capability          0x69b908930bc00801

bash-3.00# fmdump -eV | more
TIME                           CLASS
Mar 14 2011 07:56:05.511570300 ereport.io.ddi.fm-capability
nvlist version: 0
        class = ereport.io.ddi.fm-capability
        ena = 0x5eb167dc00400401
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = dev
                device-path = /
        (end detector)

        dvr-name = fp
        __ttl = 0x1
        __tod = 0x4d7d9fe5 0x1e7df17c

Mar 14 2011 07:56:05.511570350 ereport.io.ddi.fm-capability
nvlist version: 0
        class = ereport.io.ddi.fm-capability
        ena = 0x5eb167dc64400801
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = dev
                device-path = /
        (end detector)

        dvr-name = fp
        __ttl = 0x1
        __tod = 0x4d7d9fe5 0x1e7df1ae

Mar 14 2011 07:56:06.048252100 ereport.io.ddi.fm-capability
nvlist version: 0
        class = ereport.io.ddi.fm-capability
        ena = 0x5eb367b0e2000001
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = dev 

After lot of Investigation i found this is a known bug on solaris. As Sun Support is under Oracle umbrella and they have migrated sunsolve.com to myoraclesupport, still SR service for sun not working.


We have to Install patch 141874-09 to overcome this issue. This patch is a core kernel patch so need to restart mahcine and perform a full root FS backup.

bash-3.00# uname -a
SunOS testdb 5.10 Generic_141444-09 sun4u sparc SUNW,Sun-Fire-V440


bash-3.00# patchadd -p | grep 141874-09
bash-3.00#

Installing this patch:

bash-3.00# patchadd 141874-09
Validating patches...

Loading patches installed on the system...

Done!

Loading patches requested to install.

Done!

Checking patches that you specified for installation.

Done!


Approved patches will be installed in this order:

141874-09


Checking installed patches...
Executing prepatch script...
Disabling kernel module unloading ...

Verifying sufficient filesystem capacity (dry run method)...
Installing patch packages...

Patch 141874-09 has been successfully installed.
See /var/sadm/patch/141874-09/log for details
Executing postpatch script...

Patch packages installed:
  SUNWcfcl
  SUNWcfclr
  SUNWfchba
  SUNWfcip
  SUNWfcmdb
  SUNWfcp
  SUNWfcsm
  SUNWfctl

bash-3.00#

After installation of patch restart the server with reboot -- -r option.

This should solve the issue and shouldn't increase logs under /var/fm/fmd.

Happy Reading :)