Lots of NFS cross mounts?

[Karl Kleinpaste]

berger at datacube.UUCP writes:
   We are planning to have about 30 Sun 3/50's and 3/60's on our network.
   Our plan is to put 150Mbyte to 380Mbyte SCSI drives on all of these
   instead of running them diskless. It seems to be cheaper than having a
   few large nd servers. Personal and some group file systems would be on
   each of these local disks and they would generally be cross NFS
   mounted all around the net.

   Does anyone have any experience with having many (30 or more) partions
   cross mounted on many (30 or more) machines? Are there any impacts
   that we should be aware of?

We're working in that vicinity.  My particular 3/50 has 32 NFS
filesystems mounted, plus the ND partitions / and /pub.  Our
partitions are physically resident on a mix of Sun3/180 fileservers
and Pyramids.  We have been pretty nervous about moving to soft mounts
everywhere; a lot of programs behave quite well in the presence of
write I/O failures, but a lot don't, and we have to worry quite a bit
about confused undergrads wondering what Emacs is trying to tell them
when the message "file disappeared; hope that's OK" (or whatever the
actual text is; I forget just now) occurs in the minibuffer.  Shell
">" redirection is particularly uninformative when it can't
successfully open the file, but again I can't remember the exact
diagnostic - but it does not intuitively connect to a missing server.
Nonetheless, on the subnet where the staff Suns live, we have most
everything soft-mounted and it seems to be doing really quite well.
I've been advocating it for some months and we will probably get there
over summer.

One of the bigger problems to explain to people who do not fully grok
your network is why they can't get at their files when particular
servers are missing.  Our network is intended to be totally uniform:
every single 3/50 can be logged into by anyone who can login to any of
them, and an identical view of the world is presented on all of them.
We on the staff do personalized diddling (my desk 3/50 mounts 6
filesystems more than normal, for example), but in general one can
expect this total uniformity.  Now, a user whose files live on the Sun
server Fish doesn't understand immediately what's wrong when he can
login to a 3/50 named Ostrich (on the Bird subnet) but can't see the
files in his home directory when Fish is down.  It would have been
much more obvious that there was a problem if he'd tried to login on
Carp, since, as an ND client of Fish, he wouldn't have gotten anywhere
at all.  This sort of partial success is what drives our more naive
users bananas (not to mention the operators and consultants who are
answering questions).

Sooner or later, we're going to run into the minor headache that the
kernel's mount table will be too small to let us mount as many
filesystems as we want.  I don't remember just now where that can be
config'd, but the procedure exists and we have to plan on putting it
to use before too long.  You will have to do that, too.

In the realm of subjective personal opinion, I think your plan for
large numbers of discs spread across more-or-less personal machines is
an administrative nightmare.  We've found that the centralized server
approach is relatively lacking in pain to manage, whereas if we had
local discs on a large fraction of the systems here, we would never
get any real work done at all.  As it is, with 11 Sun fileservers
(roughly 12-20 3/50 clients apiece) and 3 Pyramids (soon to be 5),
major client reconfigs are something we delay until breaks between
academic quarters in the hope of not screwing up the work of too many
people at once.  Software updates can be simplified some via rdist and
similar tools, but still one must wonder about the sanity of that many
discs spread across that many systems over which I expect you don't
have full physical access control.  (Our servers live in protected,
locked machine rooms, of course.)  If you want to go that route, feel
free, but I don't envy you the task.

In any event, if little 3/50s are going to be providing disc service
to substantial numbers of other systems, your CPUs are going to be
screaming for relief.  At least try to do it all with 8Mb 3/60s lest
you be stuck with 4Mb 3/50s that spend all their time swapping.  One
advantage of lots of local disc is that you'll be able to swap to disc
rather than over the ethernet, but I would consider that small
consolation against the unavoidable overhead of that much NFS traffic.

--Karl