Re (long): Optical Jukebox Page Image Databases on Unix?

Sat Dec 15 01:15:45 AEST 1990

In article <1990Dec3.142714.20868 at watdragon.waterloo.edu> tbray at watdragon.waterloo.edu (Tim Bray) writes:
>There is now good optical disk jukebox technology; a common application is the
>storage of millions (literally) of compressed page images; essentially just an
>optical file cabinet.  Group 4 compressed Tiff format is quite common.
>
>Recently went looking for somebody who's doing this on a Unix box; shouldn't
>be hard, these things have SCSI interfaces.  Was rather surprised to find
>nothing out there.

But here we are!!

>Anybody doing this?  Better yet, anybody selling tools to automate and control
>this process?  Anybody at least have a usable driver interface?  Ideally for
>DEC RISC boxes, but any popular system would be of interest.

OK. Here's a story about our product. It is rather lengthy, but it
is as dense as possible and still clear for a relatively broad
audience.

------------------------------
INTRODUCTION

The Philips Megadoc Optical File Server is a server that offers
a very large ON LINE storage capacity in NFS networks.

It uses Optical storage technology to achieve that, combined with
jukebox support, transparent volume  management and a file
system. The Optical Disk File System takes WORM characteristics
fully into account and acts like a normal magnetic disk based
file system that has rewritable files and directories.

The Optical File Server itself however, does not make any
assumptions on the networking interfaces used, so it can be used
with any networking solution that is offered by the platform it
is running on.

PORTABILITY

The Optical File Server is portable. Several UNIX platforms are
supported. It even runs on a 386 AT platform at the low end!

The file server is a software product, that can be sold with
hardware to end-users, but does in no way rely on special
hardware support. It does not depend on devices or platforms that
have been specifically adapted for this optical file server, but
runs on standard available hardware.

COMPONENTS

The file server software consists of three main components:

     ODFS, the optical disk file system
     VM, transparent volume management
     device drivers

The ODFS takes care of files, caching on magnetic disk and
recovery features. Its user interface is the UNIX file interface
and behaves to the user as any normal rewritable magnetic disk
based file system. Volume Management takes care of transparently
loading the actual disks on drives when requested. It also
controls the jukebox when present. Device drivers implement
device dependent functionality, to make devices available for the
other software layers. Drivers included in the optical file
server product are drivers for optical disks and jukeboxes.

These main components are independent of each other. This means
that the drivers can be used without the rest of the software,
the ODFS only needs the optical disk driver(s) to be present and
VM only needs the jukebox driver. ODFS can be used on systems
without VM. VM can be used on systems without ODFS.

ODFS: PORTABILITY

The optical disks file system is implemented using standard UNIX
in-kernel interfaces, such as the File System Switch in UNIX
System V.3.2 and VFS in SunOS. It is therefore easily portable
to other UNIX flavours. UNIX System V.4 is planned for 1991.

ODFS: FLEXIBLE CACHING AND TUNING

ODFS makes extensive use of caching, both in core as well as on
magnetic disk. A magnetic disk partition is always assigned to
one or more Optical Disk File Systems. The system can be set up
in a very flexible way by assigning magnetic disk partitions of
various sizes to groups of optical disk file systems. This allow
the system administrator to tune the server in such a way that
the most frequently used optical disk file systems have the
largest cache space.

This kind of tuning is not required when absolute top performance
in not an issue; the system will automatically balance on the
optimal use of cache space, depending on the requests that have
been made to it.

ODFS: RECOVERY

The recovery of the caching system is fully supported in case of
power failures etcetera. A lot of effort has been put in the
ability to reconstruct the cache from the optical disks to
protect the user against magnetic disk crashes.

ODFS: TRANSPORTABILITY

Optical disks containing file systems can be transported to other
servers easily, since optical disks can be handled as
independent, self contained, media. They contain complete file
systems that do not rely on other optical disks, thus preventing
mistakes that would be made when a complete set of disks would
have to be transported in a fixed order.

ODFS: STANDARD USER INTERFACE

Since the Optical Disk File System completely adheres to standard
UNIX kernel interfaces, there is no special interface to access
files. ODFS behaves like any standard UNIX file system. Once
mounted, files can be accessed with the normal UNIX system calls.
This means that all applications using the standard system calls
will run on ODFS without change. This is also true for the UNIX
commands itself. ODFS comes with a small number of specific
utilities for maintenance and setup.

ODFS: INDEPENDENT

The ODFS can be used independent of volume management, but it
relies on at least one optical disk driver being present. ODFS
has no dependencies on the magnetic disk subsystem, as long as
it is present.

VOLUME MANAGEMENT: TRANSPARENCY

The Optical File Server is meant to operate in an environment
where large volumes of data have to be stored on optical disks.
As optical disks are removable media, there is no fixed relation
between the disk and the drive on which the disk is loaded, like
there is for magnetic disks. As a consequence, it is not possible
for application programs to access the physical drive, because
they have no way to be sure that the right disk is loaded on it
during the accesses.

To overcome this, Volume Management offers the concept of volume
devices. When application programs access these volume devices,
i.s.o. the physical devices, Volume Management takes care that
the corresponding disk is loaded on a drive and routes the access
to the selected drive. In this way Volume Management offers
complete location transparency.

Volume Management supports stand-alone drives and drives
contained in jukeboxes. In the first case, operator action is
required to load/unload disks; in the second case the
loading/unloading is performed completely automatic in a way
which is optimized towards high throughput and low response
times.

When Volume Management is combined with the Optical Disk File
Systems, applications can access files on mounted file systems.
ODFS will use a magnetic disk cache as much as possible, but when
an actual I/O to Optical Disk is needed, Volume Management will
automatically load the requested disk.

DEVICE DRIVERS: STANDARD INTERFACES

The device drivers supplied with the file server are the software
modules that allow the rest of the system to work with the
specific optical filing peripherals. They completely obey the
standard UNIX interfaces for drivers. Access to drivers is
possible via standard UNIX nodes in /dev. Device drivers are
available for a wide range of devices. To support other devices,
new device drivers can be easily added and integrated into the
system. The device drivers may be used independent of the other
software components of the optical file server.

HARDWARE: OPTICAL DISKS

Several Optical disk drives are supported, currently all from
LMSI:
     LD510     5.25" WORM
     LD520     5.25" Multi Function (shortly)
     LD1200    12" WORM 2GB
     LD4100    12" WORM 5.6GB (shortly)
     LF4500    12" WORM Rapid changer 5x5.6GB = 28GB (shortly)

The LF4500 is a device constructed around the LD4100 drive. It
is a disk changer containing 5 disks, but it acts like a single
device. It is in fact a small 12" jukebox with a single drive as
an integral part of it. The disk change times are very short as
compared to conventional jukeboxes (4s for any disk replacement) 

HARDWARE: JUKEBOXES

A number of jukeboxes are supported:
     IDE, 10 disks, 1 drive, 5.25" (shortly)
     HP C1710A (as LMSI's LF512), 32 disks, 2 drives, 5.25"
     FileNet OSAR-64, 64 disks, 4 drives, 12"
     FileNet OSAR-200, 204 disks, 4 drives, 12"
     FileNet and Cygnet are considered for the LD4100

There are no plans for support of other drives and jukeboxes, but
there are no technical objections. The hardware mix mentioned
above is what Philips selected for their own sales. The software
is layered in such a way that support of other devices is easily
added. This means that all hardware depedencies are grouped
together in as few modules as possible.

MORE INFORMATION

For more information contact me at:

     email:    fierst at idca.tds.philips.nl    or
               hp4nl!philapd!fierst
     fax:      xx31 55 43 3488
     mail:     M. Fierst van Wijnandsbergen
               OFS - BK
               P.O. box 245
               7300 AE  Apeldoorn
               the Netherlands
               Europe  
------------------------------
-- 
#  Michiel Fierst van Wijnandsbergen   Internet fierst at idca.tds.philips.nl #
#  Philips Telecomm. and Data Systems  UUCP       ...!mcvax!philapd!fierst #