Problem with compress

[Dan Bernstein]

In article <1991May15.062504.28369 at looking.on.ca> brad at looking.on.ca (Brad Templeton) writes:
> It's a tough problem, actually.   It is nice to take the software tools
> approach of splitting archiver and compressor, but this makes it more
> difficult to do some of the cute things you can do when you combine them.
> The simplest of which is the fairly standard archive of compressed
> files, with each file compressed individually and extractable without
> decompressing the entire archive -- a serious problem if it is a
> multi-media archive or very large.

This is really getting away from comp.compression into
comp.unix.programmer, but anyway: All you have to do to make this work
is define a format for encoding multiple files into one stream, then
have your compressor (and, while you're at it, all your other tools)
work on such a stream. Let's say, for instance, that you use cpio
format, and have compress -f understand that format. Then you just pipe
cpio through compress -f and poof! all the files are individually
compressed.

I don't think cpio (or tar or extended cpio) is a particularly good
format for this: it's too restricted for some tasks (e.g., when you want
to multiplex two arbitrary-length streams) and too complex for others
(e.g., when you don't need or want filenames). So I've been working on
yet another format and a set of routines for applications to use that
format. Once it's done it'll solve a lot of old problems: grepping
through compressed files, for instance, or compressing each file in an
archive individually.

---Dan