small files and big directories (was hard links to symlinks)

[t patterson]

In article <24523 at mimsy.umd.edu> chris at mimsy.umd.edu (Chris Torek) writes:
>>In article <24483 at mimsy.umd.edu> I wrote:
>>>... The ratio of small to big should suggest whether special handling
>>>for small, big, or `very big' files and/or directories would be useful.
>
>In article <1709 at cirrusl.UUCP> dhesi%cirrusl at oliveb.ATC.olivetti.com
>(Rahul Dhesi) writes:
>>What you will find will probably reflect the fact that big directories on
>>UNIX machines are already known to be undesirable ... a conclusion like
>>"big directories seldom exist, so we need not worry too much about them" is
>>natural, but probably only because it is a self-fulfilling prophecy.

this may depend on what you consider to be "big", but I can think of 2 
applications where "big" directories are produced:

1. news:
    for example, an ls -ld on /usr/spool/news/comp/unix (minus the files 
    in comp.unix) on my news server:

	   drwxrwxr-x  2 news         2560 May 28 08:08 aix
	   drwxrwxr-x  2 news         2560 May 28 00:33 aux
	   drwxrwxr-x  2 news          512 May 24 00:31 cray
	   drwxrwxr-x  2 news        16384 May 28 16:16 i386
	   drwxrwxr-x  2 news         1024 May 27 23:33 microport
	   drwxrwxr-x  2 news        18432 May 28 17:20 questions
	   drwxrwxr-x  2 news         6144 May 28 10:07 ultrix
	   drwxrwxr-x  2 news        11776 May 28 17:20 wizards
	   drwxrwxr-x  2 news         9728 May 28 14:40 xenix

    comp.unix.{i386,questions,wizards,xenix}, for example, are getting big
    to the point of unwieldy. (yeah, we could shrink them by rebuilding
    the directories, but that's pretty awkward.) the others aren't 
    exactly scrawny either.

2. mh-based mail
     we have more than a few users who have thousands of files (messages)
     per folder; this can be really slow. moreover, when you have been
     accumulating mail for a decade, it is _easy_ for some people to get
     themselves in this kind of bind.

In article <24523 at mimsy.umd.edu> chris at mimsy.umd.edu (Chris Torek) writes:
>Perhaps---but I would note the following:
>
> a. Personally, I prefer to keep my own directories relatively small.
...
> b. As a programmer, I prefer to keep my programs' directories relatively
>    small, ...
...
>All in all, though, I am still unconvinced that adding a special case
>for big directories would help overall.

   well, I would agree that

      a. the problem needs more study -- how can we quantify how much
	 impact the current directory setup has on overall usage?

      b. most naive people learn about directories eventually; in general,
	 people find it unwieldy to try and manage directories with 
	 thousands of entries. more sophisticated people also seem to
	 avoid this problem.

   but

      when the application (news, mh) hides some of that complexity from
      you, it is easy for things to get out of hand.

  maybe the application needs fixing, maybe it's the filesystem.

  (this thread is kinda curious to me, because it stirs memories of an
old Chris Torek posting which I think stated "the filesystem IS the
database" and discussed the advantages of using hashing to build directory
entries.)
--
t. patterson		domain: tp at wsl.dec.com    path: decwrl!tp
            		enet:	wsl::tp
     			icbm:	122 9 41 W / 37 26 35 N