Root File System Invisible File(s)

[Romain Kang]

This sounds like some program has opened a temporary file and unlinked
it so that it does not appear in the filesystem, while holding the
descriptor open.  I'm not aware of any standard OSx utilities that do
this in the root file system (other than those which use /tmp, and you
already said /tmp was a separate partition); perhaps it may be a local
application.  Your description suggests that there is one large file
holding the space, though you didn't say anything about getting inode
counts from df.

To find the open file, I would use "pstat" and "fstat".  (The latter
is not a standard part of OSx but you should be able to find it at a
reasonably well stocked public Pyramid source archive; I can send a
copy if you don't have it or time to go rummaging.)

"pstat -i" will show the active inodes on the system; look for inodes
with large sizes on your root device (8,0 if pdisk00a).  Next, weed out
the known big files (/vmunix, /etc/.*tmp) according to inode.  If you
don't see any obvious culprits, then getting fstat won't help.  If you're
feeling brave, you can try using "find" with -inum if you can limit the
search to directories that don't include mount points.  Chances are you
won't find the big file this way, since you've looked for it already
with the "normal" methods.

At this point, you can run "fstat {rootdev}", where {rootdev} is the
block device used to mount your root partition.  The output will look
something like this:
USER     CMD          PID   FD DEVICE  INODE    SIZE TYPE  NAME
root     swapper        0   wd  8,  0      2   10240 dir   /dev/iop/pdisk00a
root     init           1 text  8,  0   2948   36864 reg   /dev/iop/pdisk00a
root     init           1   wd  8,  0      2   10240 dir   /dev/iop/pdisk00a
root     init           1   12  8,  0   4118       0 chr   /dev/iop/pdisk00a
 :
 :
romain   fstat       9892    0  8,  0   2879       0 chr   /dev/iop/pdisk00a
romain   fstat       9892    1  8,  0   2879       0 chr   /dev/iop/pdisk00a
romain   fstat       9892    2  8,  0   2879       0 chr   /dev/iop/pdisk00a
romain   fstat       9892    3  8,  0   4797       0 chr   /dev/iop/pdisk00a
romain   fstat       9892    4  8,  0    723 1232896 reg   /dev/iop/pdisk00a
					    [^^^^^^^ umm, pyramid:/vmunix...]

You can match the inode number in this listing with the one from pstat
to find the user and the process holding the file open (if that is
indeed the cause of your space problem).  Note that you could run fstat
alone, and not bother with pstat at all if you already have fstat.

Another classic misoperation is for users to write to a tape device
that isn't in the file system (e.g., /dev/rmt9 when rmt0 was intended);
user then notices error and deletes file.  This probably isn't you're
problem, since (1) the user would have to be root, and I assume your
root users are known and responsible, and (2) if user X catches the
error, I don't think he would repeat the same mistake so frequently.
--
"Eggheads unite!  You have nothing to lose but your yolks!"  -Adlai Stevenson