Disk striping experiences (summary)

[David E. Bernholdt]

Thanks to all who responded to my query about striping disks.  Here is
the promised summary...

Machines mentioned include a preponderance of Convexen, also Alliant,
Apollo, and Cray.  Most experiences were positive.

Problems mentioned include:

1) Disk crashes/bad disks.  These are handled like any other file
system for the most part -- you back them up regularly and they're
fine.  One person said they had a string of bad disks which was bad
enough to put them off of striping.

2) Don't put multiple partitions in the same disk in a striped
partition -- the disk will have to seek all over the place!

3) Due to limitations of unix file systems, it seems that on many
implementations, striped partitions are limited to 2GB total size.
Presumably some vendors have ways around this, but I don't claim to
know who does & who doesn't -- ask the sales people.

4) One person mentioned that on the Alliant, the file system manager
requires an extra 10-12 Mb to manage striped file systems.
Apparently, this is not well publicized/documented, and obviously it
could be a problem on a relatively small-memory machine.  This was not
mentioned by anyone else, so I don't know how many other vendors may
have similar "problems".

5)  Having read through the FPS documentation (what little there is)
about striping, I found the interesting restriction that the number of
disks striped into a given partition must be a power of two.  No one
mentioned this kind of restriction for other machines -- and Convex
owners mentioned 3-way and 6-way striping, so they obviously don't
have any such limitation.

Performance issues:

No one reported any kind of before/after benchmarks, but people seemed
uniformly pleased by the performance improvement.  One person says
they got nearly a 4x improvement on I/O rate for a 4-way stripe.

Obviously, striping is only going to help for systems which are I/O
bound.  Note that that's systems, and not necessarily just particular
applications -- if a potentially I/O-bound application has to compete
against enough low-I/O work already, faster I/O may not matter).
Also, striping really only helps for large files -- small ones do just
as well on regular file systems.

One Convex owner said they had a setup with 5 disks on two controllers
-- three regular and two striped together (on separate controllers).
They were warned that having the non-striped disks served by the same
controllers would create a bottleneck, however, they have found it not
to be so.  They are running XY451D controllers and Fujitsu Eagle
disks.  You definitely need to have the striped disks on different
controllers, though.

In conclusion:

One person said explictly that the general fear of striping was pretty
much unfounded, as long as you have reliable hardware and treat the
striped file systems as you would others -- regular backups of
important data, etc.  From what I've seen so far, I'm inclined to
agree.  Hopefully, after we stripe our file systems, I'll still
agree...

Thanks again to everyone who responded.
-- 
David Bernholdt			bernhold at qtp.ufl.edu
Quantum Theory Project		bernhold at ufpine.bitnet
University of Florida
Gainesville, FL  32611		904/392 6365