[SUMMARY:]/var/full

From: zhouna (zhouna@asiainfo.com)
Date: Sun May 30 1999 - 21:31:13 CDT


my original question is:

one my E3500 server, df and du give different result on /var:

# du -ks /var
39853 /var

# df -k /var

/dev/vx/dsk/var 482824 482581 0 100% /var

Many people think that there is a file that was deleted that is still
being held open by some process. they really hit the nail on the head.

when i killed all the cron related processed, the problem is solved,the
space is freed.

The fact is it is not me who deleted a file that was still being opened
by other processes,it is the solaris itself,in my opinion.

solaris uses "mv $LOG $OLOG" in /etc/cron.d/logchecker to update cron
log. Maybe when mv is runed, /var/cron/log is being written.*I'd like to
get more suggestion on it.*

thanks to everyone who give the answer.especially to Thompsom Dave who
sent me some useful materials. I include it in the attachment to share
it with everyone.

Zhou Na

Subject: FW: df verses du
Date: Fri, 28 May 1999 08:54:27 -0400
From: "Thompson, Dave" <ThompsD1@bsci.com>
To: zhouna@asiainfo.com

----------
From: Herman, Yvonne
Sent: Friday, October 23, 1998 11:39 AM
To: Keefe, Matt; Thompson, Dave; Brown, Howard; Nichols, Jason; Herman,
Yvonne
Subject: df verses du

Info Docs article 17509 [ Notify of changes ]
[ Edit/Retrieve Marked Documents ] [ Mark Document ]
Did the search result in the document you were looking for? Send Feedback

------------------------------------------------------------------------

INFODOC ID: 17509

SYNOPSIS: Inodes and the filesystem
DETAIL DESCRIPTION:

Inodes and the filesystem:
  1. one explanation for why du and df can disagree
  2. how deleting a file may not free up disk space
  3. how can I be out of inodes?
  4. a diagnostic procedure
  5. a diagnostic script

1. one explanation for why du and df can disagree
-------------------------------------------------
df describes disk usage
du adds up the size all the files on a filesystem
They can report wildly different figures.
One reason:
  processes write to files
  files can be deleted while processes are writing to files
  so a file may be gone, but its inode may not be freed up
  example:
  # cat > bogus
  testing
  testing
  ^Z
  Stopped (user)
  # ls -i bogus
        2921 bogus
  # fuser bogus
  bogus: 14947o
  # rm bogus
  # ls -i bogus
  bogus: No such file or directory
  # /usr/proc/bin/pfiles 14947
  14947: cat
    Current rlimit: 64 file descriptors
     0: S_IFCHR mode:0620 dev:32,120 ino:211552 uid:0 gid:7 rdev:24,17
        O_RDWR
     1: S_IFREG mode:0644 dev:32,5 ino:2921 uid:0 gid:1 size:16
        O_WRONLY|O_LARGEFILE
     2: S_IFCHR mode:0620 dev:32,120 ino:211552 uid:0 gid:7 rdev:24,17
        O_RDWR
  #
  This example shows how a file can be deleted but still take up room
    on a hard drive:
    A file (bogus: inode 2921) is created by a process (pid: 14947)
    The files is deleted and it is demostrated that it no longer resides
      on the file system.
    But pfiles shows the inode is still open and is taking up 16 bytes
      of disks space

2. how deleting a file may not free up disk space
-------------------------------------------------
This is the same reason a disk may be full and removing a file does not
  clear space.
If you do a fuser on a file before removing it you will determine if
  removing it will do any good.
Example:
  # ls -l /var/adm/messages
  -rw-r--r-- 1 root other 218611 Jul 29 02:18 messages
  # fuser /var/adm/messages
  messages: 164o
  # ps -ef | grep 164
      root 164 1 0 22:47:49 ? 0:03 /usr/sbin/syslogd
  # kill 164
  # rm /var/adm/messages
  # touch /var/adm/messages
  # /usr/sbin/syslogd
  # ls -l /var/adm/messages
  -rw-r--r-- 1 root other 7592 Jul 29 11:46 messages
  #
  This example demostrate a way to remove the /var/adm/messages and
    actually clear disk space.
  Notice how the messages file jumps up to 7k after syslog is restarted.
  syslog will dump the contents of the dmesg buffer into the
    /var/adm/messages when it is started.

3. how can I be out of inodes?
------------------------------
Not all inodes captured by a processes are normal files.
Many are character devices which take no disk space but will remove a
  inode from possible use.
These character devices may take up memory which in turn reduce the
  amount of memory available to your tmpfs file systems (ie, /tmp).
Example:
  # /usr/proc/bin/pfiles 10234
  10234: xterm
    Current rlimit: 64 file descriptors
     0: S_IFCHR mode:0620 dev:32,120 ino:211538 uid:52475 gid:7 rdev:24,3
        O_RDWR|O_LARGEFILE
     1: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
        O_WRONLY|O_LARGEFILE
     2: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
        O_WRONLY|O_LARGEFILE
     3: S_IFCHR mode:0666 dev:32,120 ino:210791 uid:0 gid:3 rdev:13,12
        O_RDWR
     4: S_IFIFO mode:0666 dev:171,0 ino:1626250456 uid:0 gid:0 size:0
        O_RDWR|O_NONBLOCK FD_CLOEXEC
     5: S_IFCHR mode:0000 dev:32,120 ino:65232 uid:0 gid:0 rdev:23,13
        O_RDWR|O_NDELAY
  # ps -ef | grep 10234
    dsweet 10234 10220 0 23:16:58 ?? 0:01 xterm
  # ls -lL /dev/dsk/c0t0d0s0
  brw-r----- 1 root sys 32,120 Jun 24 19:51 /dev/dsk/c0t0d0s0
  # df /
  / (/dev/dsk/c0t0d0s0 ): 1097712 blocks 456994 files
  # find / -mount -inum 211538
  /devices/pseudo/pts@0:3
  # find / -mount -inum 210782
  /devices/pseudo/cn@0:console
  # find / -mount -inum 65232
  #
  This example shows that dsweet's xterm has captured 3 inodes* on his
    root partition but none were normal files.
  None of them have size so they are special character files (a raw
    device).
  The last inode (65232) was not found on the file system.
  Therefore inode 65232 can not be freed unless its process lets go of
    it or the process is killed.

  * inode 210782 is the console and it was captured twice

4. a diagnostic procedure
-------------------------
This is all well and good, but how does a user figure out where all
  his inodes have gone?
The procedure:
  Determine the major and minor numbers for the troublesome device:
    # df /
    / (/dev/dsk/c0t0d0s0 ): 1097710 blocks 456994 files
    # ls -lL /dev/dsk/c0t0d0s0
    brw-r----- 1 root sys 32,120 Jun 24 19:51 /dev/dsk/c0t0d0s0
    #
  Determine which inodes on the device are opened by processes:
    # ls /proc
    0 10220 10235 10251 10282 10386 136 160 227 298
306
    1 10222 10236 10253 10300 10429 141 178 235 3
3060
    10120 10227 10237 10255 10310 10446 143 184 236 301
308
    10130 10228 10238 10266 10314 10484 14778 194 267 302 99
    10134 10229 10239 10268 10338 10485 14934 2 270 303
    10144 10230 10240 10270 10352 10567 14935 2115 275 304
    10167 10231 10241 10272 10353 106 14937 2117 285 305
    10169 10232 10242 10276 10364 10605 14947 212 288 3055
    10170 10233 10243 10278 10366 108 14969 215 289 3057
    10219 10234 10245 10280 10385 119 15008 225 295 3058
    # /usr/proc/bin/pfiles 0 | grep "dev:32,120"
    # /usr/proc/bin/pfiles 10220 | grep "dev:32,120"
    0: S_IFCHR mode:0620 dev:32,120 ino:211538 uid:52475 gid:7 rdev:24,3
    1: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
    2: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
    3: S_IFCHR mode:0666 dev:32,120 ino:210791 uid:0 gid:3 rdev:13,12
    8: S_IFCHR mode:0000 dev:32,120 ino:62920 uid:0 gid:0 rdev:42,126
    9: S_IFCHR mode:0000 dev:32,120 ino:55000 uid:0 gid:0 rdev:41,171
    10: S_IFCHR mode:0000 dev:32,120 ino:64688 uid:0 gid:0 rdev:42,130
    11: S_IFREG mode:0644 dev:32,120 ino:313393 uid:0 gid:3 size:316
    #
    Just by checking the first 2 processes I found 7 unique processes
      that are open on the file system.
    You have to repeat this step for all the processes found in
      /proc to get all the open inodes on the file system.
  Determine if the inodes exists on the file system:
    # find / -mount -inum 211538
    /devices/pseudo/pts@0:3
    # find / -mount -inum 210782
    /devices/pseudo/cn@0:console
    # find / -mount -inum 210791
    /devices/pseudo/mm@0:zero
    # find / -mount -inum 62920
    /usr/lib/lpshut
    # find / -mount -inum 55000
    # find / -mount -inum 64688
    # find / -mount -inum 313393
    /etc/group
    #
    Therefore only 2 of the 7 inodes captured by process 10220 are rogue.
    Rouge is my way of describing an open inode that doesn't have
      cooresponding file in the file system.
    You would have to repeat this step for all the inodes found above to
      see which open inodes are rogue.
  Once done with the last step you can go back to see how many inodes
    are rogue, if they are special character devices, and how much space
    they are taking up.

5. a diagnostic script
----------------------
  There has to be an easier way then the procedure just described.
  You could write a script to do the work for you or use mine:
--------------------------------------------------------------
#!/sbin/sh

tmpfile=/tmp/icheck.$$.tmp

if [ "$1" ]
then
  device=$1

  if [ -b "$device" ]
  then
    mntpoint=`/usr/sbin/df | /usr/bin/grep $device | /usr/bin/awk ' { print
$1
} '`

    if [ $mntpoint ]
    then

      firsthalf=`/usr/bin/ls -lL $device | /usr/bin/awk -F\, ' { print $1 }
'`
      secondhalf=`/usr/bin/ls -lL $device | /usr/bin/awk -F\, ' { print $2 }
'`

      major=`/usr/bin/echo $firsthalf | /usr/bin/awk ' { print $5 } '`
      minor=`/usr/bin/echo $secondhalf | /usr/bin/awk ' { print $1 } '`

      possiblePIDs=""
      for pid in `/usr/bin/ls /proc`
      do
        inodes=`/usr/proc/bin/pfiles $pid 2> /dev/null | /usr/bin/grep
"dev:${major},${minor}" | /usr/bin/awk ' { print $5 } ' | /usr/bin/awk
-F: ' { print $2 } ' | /usr/bin/sort | /usr/bin/uniq`

        if [ "$inodes" ]
        then
          possiblePIDs="$possiblePIDs $pid"
        fi

        for inode in $inodes
        do
          /usr/bin/echo $inode >> $tmpfile
        done
      done

      if [ -f $tmpfile ]
      then
        inodes=`/usr/bin/sort $tmpfile | /usr/bin/uniq`
        /usr/bin/rm $tmpfile
      fi

      inum=0
      for ino in $inodes
      do
        inum=`/usr/bin/echo $inum + 1 | bc`
      done
      /usr/bin/echo $inum open inodes found on $device

      binum=0
      inum=0
      badinodes=""
      for ino in $inodes
      do
        /bin/printf "\r%d inodes found without files on file " $binum
        /bin/printf "system (%d searched)" $inum
        filename=`/usr/bin/find $mntpoint -mount -inum $ino 2>/dev/null`
        inum=`/usr/bin/echo $inum + 1 | bc`
        if [ "$filename" ]
        then
          /usr/bin/echo do nothing > /dev/null
        else
          badinodes="$badinodes $ino"
          binum=`/usr/bin/echo $binum + 1 | bc`
        fi
      done
      /bin/printf "\r%d inodes found without files on file " $binum
      /bin/printf "system (%d searched)\n" $inum

      for badino in $badinodes
      do
        /bin/printf "the following processes have captured rogue "
        /bin/printf "inode %s" $badino

        notfound="1"
        firsttime=1;
        for pid in $possiblePIDs
        do
          response=`/usr/proc/bin/pfiles $pid 2>/dev/null | /usr/bin/grep
"ino:${badino}"`
          if [ "$response" ]
          then
            notfound=""
            size=""
            if [ $firsttime ]
            then
              firsttime="";
              havesize=`/usr/bin/echo $response | /usr/bin/grep
"size:"`
              if [ "$havesize" ]
              then
                size=`/usr/bin/echo $havesize | awk ' { print $8 } '`
              fi

              if [ $size ]
              then
                /usr/bin/echo " (${size}):"
              else
                /usr/bin/echo ":"
              fi
            fi

            /usr/bin/echo " $pid"

          fi
        done

        if [ $notfound ]
        then
          echo ":"
        fi

      done

      exit 0

    else
      /usr/bin/echo error: $device is not mounted
      exit 1
    fi

  else
    /usr/bin/echo usage: icheck.sh \<device_file\>
    /usr/bin/echo error: $device is not a block special file
    /usr/bin/echo example: icheck.sh /dev/dsk/c0t0d0s0
    exit 1
  fi

fi

/usr/bin/echo usage: icheck.sh \<device_file\>
/usr/bin/echo example: icheck.sh /dev/dsk/c0t0d0s0
exit 1
--------------------------------------------------------------
Sample output from icheck.sh being run on the file system first
  shown in section 1's example:
  # /usr/local/sbin/icheck.sh /dev/dsk/c1t0d0s5
  12 open inodes found on /dev/dsk/c1t0d0s5
  1 inodes found without files on file system (12 searched)
  the following processes have captured rogue inode 2921 (size:16):
    14947
  #

PRODUCT AREA: System Administration
PRODUCT: Disk admin
SUNOS RELEASE: Solaris 2.5
HARDWARE: any

------------------------------------------------------------------------
Home | Free Services | Contract Services | Account Services | Table of
Contents
Comments about SunSolve | Help | SunSolve FAQ's | Privacy Policy

Questions or comments regarding this service? webmaster
Copyright 1994-1998 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto,
CA 94303 USA.
All rights reserved.

Yvonne Herman
Unix System Administrator
Andersen Consulting/Boston Scientific Corporation
hermany@bsci.com
508-652-5226

Subject: FW:
Date: Fri, 28 May 1999 08:54:46 -0400
From: "Thompson, Dave" <ThompsD1@bsci.com>
To: zhouna@asiainfo.com

----------
From: Herman, Yvonne
Sent: Friday, October 23, 1998 11:47 AM
To: Keefe, Matt; Brown, Howard; Thompson, Dave; Nichols, Jason; Herman,
Yvonne

Symptoms and Resolutions article 2131 [ Notify of changes ]
[ Edit/Retrieve Marked Documents ] [ Mark Document ]
Did the search result in the document you were looking for? Send Feedback

------------------------------------------------------------------------

SRDB ID: 2131

SYNOPSIS: How to increase inodes when inode table is full

DETAIL DESCRIPTION:

How do you increase inodes when the inode table is full?

SOLUTION SUMMARY:

There are two types of inodes: kernel and disk. You should be able to tell
which inode table is full by the message.

        Kkernel inode table that is full:

                        vmunix: inode table full

        Verify this by running pstat -T:

                        # pstat -T
                          301/582files
                          199/199 inodes
                          67/138 processes
                          34208/43012 swap

        Notice that the line for inodes shows that 199 are
        being used and the maximum is also 199.

        If the disk inode tabel is full, this message appears:

                        /:inode table full

        Verify this by running df -i -t 4.2:

                # df -i -t 4.2
                Filesystem iused ifree %iused Mounted on
                /dev/sd0a 2794 2794 100% /
                /dev/sd0g 7007 29857 19% /usr
                /dev/sd2h 301 11219 3% /var
                /dev/sd2a 30 9954 0% /home
                /dev/sd2g 4656 16848 22% /export

        Notice that the root partition has used all the available inodes.

        If the disk inode table is full:

              - do a dump of the filesystem, as it will have to be
                wiped out.

              - run newfs -N on the raw disk partition:

        # newfs -N /dev/rsd2g
        /dev/rsd2g: 92190 sectors in 439 cylinders of 6 tracks, 35
sectors
                47.2MB in 28 cyl groups (16 c/g, 1.72MB/g, 768 i/g)
        super-block backups (for fsck -b #) at: ....

         Notice that this partition has 768 inodes per cylinder group.

              - Make sure the partition is not mounted (if root or
                /usr, then this will have to be done from miniroot)

              - newfs -i xxx /dev/r<partition>(wherexxx is the
                number of inodes per cylinder group and
                <partition> is sd0h, sd1a, etc.)

              - Restore the file system from dumps.

             NOTE: Prior to 4.1, a maximum of 2048 inodes per
             cylinder group existed. To get around this, run:

                        newfs -c

           and increase the number of cylinder groups (increasing the
           number of cylinder groups allows more inodes in the partition).

             If kernel inode table is full, do ONE of the following:

             NOTE: the instructions on configuring a new kernel can
             be found in the System and Network Aministration manual.

             1) Edit the kernel configuration file. Increase the
                MAXUSERS value (rule of thumb: double the current
                value). Make and reboot the new kernel.

             2)Run config on the kernel configuration file. Then:

                        - cd /sys/<arch>/conf
                        - cd /sys/<arch>/<kernel>

                (where <kernel> is the name of the configuration file
                and <arch> is the kernel architecture.)

                        - edit param.c
                        - search for the line thatcontains "ninode"

                This line is the formula that calculates the number
                of kernel inodes.

                        - adjust this formula by increasing some of the
values.

                        - do a "make" on the kernel, then reboot
                          the new kernel

BUG REPORT ID: 0
PATCH ID: n/a
PRODUCT AREA: Kernel
PRODUCT: Config
SUNOS RELEASE: 4.1, 4.1.1
UNBUNDLED RELEASE: n/a
HARDWARE: All

------------------------------------------------------------------------
Home | Free Services | Contract Services | Account Services | Table of
Contents
Comments about SunSolve | Help | SunSolve FAQ's | Privacy Policy

Questions or comments regarding this service? webmaster
Copyright 1994-1998 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto,
CA 94303 USA.
All rights reserved.

Yvonne Herman
Unix System Administrator
Andersen Consulting/Boston Scientific Corporation
hermany@bsci.com
508-652-5226



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:20 CDT