Oracle Solaris Guest LDOM root file system is Full

Are you stuck in same situation where your guest ldom root file system is full and VM went into the transition mode? Then this is the right technical article that will help you to overcome this issue. We encountered this issue in one of our customer environments and were able to fix it successfully. Currently there are two volumes attached to to virtual machine

DISK
    NAME         VOLUME                 TOUT ID   DEVICE  SERVER         MPGROUP
    ijrcrm5-vol1 ijrcrm5-vol1@primary-vds0      0    disk@0  primary
    ijrcrm1-vol2 ijrcrm5-vol2@primary-vds0      1    disk@1  primary

ijrcrm5-vol1 is the OS volume attached to the VM

root@IJRSVP-SOL-CDM1:/dev/zvol/dsk/rpool# zfs get volsize rpool/ijrcrm5-vol1
NAME                PROPERTY  VALUE  SOURCE
rpool/ijrcrm5-vol1  volsize   50G    local
root@IJRSVP-SOL-CDM1:/dev/zvol/dsk/rpool#

Now we need to increase the volume size from the controller domain:

root@IJRSVP-SOL-CDM1:/dev/zvol/dsk/rpool# zfs set volsize=70g rpool/ijrcrm5-vol1
root@IJRSVP-SOL-CDM1:/dev/zvol/dsk/rpool#

Login to the virtual machine in maintenance Mode:

Enter user name for system maintenance (control-d to bypass): root
Enter root password (control-d to bypass):
single-user privilege assigned to root on /dev/console.
Entering System Maintenance Mode

Jan 22 12:18:50 su: 'su root' succeeded for root on /dev/console
Oracle Solaris 11.4.49.126.2                    Assembled August 2022
You have new mail.
root@ijrpscrm-p-w2:~# df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-1    49G  17.5G          0   100%    /
rpool/ROOT/solaris-1/var
                        49G  1.54G          0   100%    /var
/devices                  0      0          0     0%    /devices
/dev                      0      0          0     0%    /dev
ctfs                      0      0          0     0%    /system/contract
proc                      0      0          0     0%    /proc
mnttab                    0      0          0     0%    /etc/mnttab
swap                  27.1G    21M      27.1G     1%    /system/volatile
swap                  27.1G      0      27.1G     0%    /tmp
objfs                     0      0          0     0%    /system/object
sharefs                   0      0          0     0%    /etc/dfs/sharetab
fd                        0      0          0     0%    /dev/fd
rpool/VARSHARE          49G  4.96M          0   100%    /var/share
rpool/VARSHARE/tmp      49G   288K          0   100%    /var/tmp
rpool/VARSHARE/kvol     49G   288K          0   100%    /var/share/kvol
root@ijrpscrm-p-w2:~#

- Here we can see there is no space in root file system Set to expand rpool to autoexpand but unfortunately it failed due to out of space as listed below:

root@ijrpscrm-p-w2:~# zpool set autoexpand=on rpool
cannot set property for 'rpool': out of space
root@ijrpscrm-p-w2:~#

Reduced the size of dumpadm file system from 16GB to 5GB and the auto expand of rpool worked:

root@ijrpscrm-p-w2:/# zfs list -t vol
NAME                               USED  AVAIL  REFER  MOUNTPOINT
rpool/VARSHARE/kvol/dump_summary  1.22M  1.08M   144K  -
rpool/VARSHARE/kvol/ereports      10.2M    80K  10.2M  -
rpool/VARSHARE/kvol/kernel_log    16.2M    80K  16.2M  -
rpool/dump                          16G   3.7M    16G  -
rpool/swap                           4G     4G   144K  -
root@ijrpscrm-p-w2:/# zfs set refreservation=5.0G rpool/dump
root@ijrpscrm-p-w2:/# zpool set autoexpand=on rpool
root@ijrpscrm-p-w2:/#
Expand the rpool volume from the guest VM:

root@ijrpscrm-p-w2:~# zpool online -e rpool c1d0
root@ijrpscrm-p-w2:~# df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-1  68.7G  17.5G      19.7G    48%    /
rpool/ROOT/solaris-1/var
                      68.7G  1.54G      19.7G     8%    /var
/devices                  0      0          0     0%    /devices
/dev                      0      0          0     0%    /dev
ctfs                      0      0          0     0%    /system/contract
proc                      0      0          0     0%    /proc
mnttab                    0      0          0     0%    /etc/mnttab
swap                  28.7G  20.9M      28.7G     1%    /system/volatile
swap                  28.7G      0      28.7G     0%    /tmp
objfs                     0      0          0     0%    /system/object
sharefs                   0      0          0     0%    /etc/dfs/sharetab
fd                        0      0          0     0%    /dev/fd
rpool/VARSHARE        68.7G  4.96M      19.7G     1%    /var/share
rpool/VARSHARE/tmp    68.7G   288K      19.7G     1%    /var/tmp
rpool/VARSHARE/kvol   68.7G   288K      19.7G     1%    /var/share/kvol
root@ijrpscrm-p-w2:~#

We are successfully able to overcome and fix the rpool issue in the guest VM. Thanks for reading and hope it helped 😊 Regards, Zaheer