File sizes > 2^32 bytes?

[A. Lester Buck]

When will common versions of Unix support very large file
sizes of >2^32 bytes?  A client is moving some massive software
packages that deal with file sizes of greater than 4 GB from
an MVS hosted environment to a workstation.  (I know it takes
a long time to just read such a file, but once loaded only
small parts are processed in a random access, interactive mode.)
I can glue physical drives together into a large filesystem at
the driver level, but there is no getting around the 32-bit data
type in inode.h .  This appears to be the limit in any Unix
implementation on 32-bit hardware, from System VR3.2 to AIX.
A recent posting mentioned that DG/UX uses 48 bits for block
numbers in the filesystem, but I couldn't tell if that included
file size.  Also, it seems VAX C has already implemented the
ANSI C typedef fpos_t, a data type that holds whatever data
is necessary to seek anywhere in a file, as a 64 bit entity.
[fpos_t is returned by fgetpos() and is input to fsetpos().]
Does VMS currently support file sizes of greater than 32 bits?
I presume that UNICOS on the Cray and other native 64 bit Unix
implementations do currently support filesizes larger than 4 GB.

Will System VR4 support such large file sizes?  I know we could
always implement our own filesystem, but virtually every utility
assumes that the file size fits in 32 bits.  And the applications
could be reworked to split the file into chunks and manage the
chunks, but that is not very attractive, either.  What does POSIX
say about the maximum file size?

On a related note, there is a public domain proposal circulating for
a data interchange filesystem on WORM disks in the geophysical
data processing industry.  This file system has a standard generic
hierarchical directory structure, lots of room for long filenames
(126 bytes), but specifies only 32 bits for file size.  And this
for 12" WORM media that by the end of this year will have 2 GB/side!
This can still be changed, but I hope the ANSI optical filesystem
committee forming next month will pick a larger data type
for file size.

It is interesting to check what people think is the maximum
file size in various operating systems.  4 GB is the typical maximum
under Unix, not withstanding those who calculate that the triple
indirect blocks can keep track of 16+ GB.  (Are there any utilities
that actually interpret the sign bit of the file size/seek position,
which would prevent using the last 2 GB of the 32 bits?)  Ask the
same question of an MVS expert and there is a significant pause,
then some mumbling about 1 primary unit and 255 secondary units, then
calculations on how much a 3380(?) tri-density drive (unit) can
currently hold (about 1.8 GB/drive, if I remember correctly).  So
the calculation is much more convoluted, but it does have the
interesting property that maximum file size tracks disk technology,
so room doesn't run out at a fixed size.  (So how, you ask, does MVS
seek within such a big file with a 32-bit data type?  It seeks
by records, of course!)  In addition, I was told that for sequential
access, MVS disks can be arranged in an unlimited sequence, with a
zero count-key indicating physical end of file.  And I heard a rumor
that IBM hit the wall with AIX/370 under VM, when they reached the VM
minidisk maximum size limit, long before the 32 bits ran out.

-- 
A. Lester Buck		...!texbell!moray!siswat!buck