Restoring incremental dumps - summary of responses

[Robert Perlberg]

<>

I'd like to thank the people who responded to my query
about restoring incremental dumps.  In brief, I complained about
the fact that I was unable to restore an incremental dump because
of restor's questionable method of using the original inode numbers
whether they were still valid or not.  Since then, I have run
into another related problem.  I found it impossible to restor
a tape dumped from a large filesystem onto a smaller filesystem,
even though the smaller filesystem was big enough to hold all
of the data, simply because the inode table did not have slots
that restor tried to write files to.

Most of you responded by posting news, but I got one interesting
mail response:

>> From: seismo!hao!stcvax!lat
>> 
>> I have on several occasions successfully restored files systems
>> from multi-level dumps.  This is supposed to be done only on
>> unmounted file systems.  All the work is done with inodes instead
>> of with file names to make the job easier and smaller, especially
>> for machines like PDP-11's which have limited addressing space.
>> 
>> The task of keeping track of multiple file links, deleting a file
>> occurence from one directory, etc., would be much more difficult
>> otherwise.

Your explanation of why restor was written so poorly is a typical
UNIX cop-out.  If the writers of restor had just put a couple
of weeks more work into it, it would have by now saved many times
that much work in reconstruction of corrupted filesystems and
re-doing of lost work.

You'll notice in the following exerpt from a reply that was posted
to unix-wizards that Berkeley restore already does what I suggested
it should.  Why doesn't everyone use Berkeley UNIX?  It seems so
superior.

>> From: rdin!philabs!seismo!harpo!utah-cs!utah-gr!thomas
>> 
>> Well, I know this isn't an answer for your case, but the 4.2 restore
>> (note the 'e' on the end) program must be run on a MOUNTED disk, and
>> creates the new files using the standard creat and mknod calls.  Really
>> helps to leave your disks in a more consistent state.  (At least, this is
>> my understanding of what it is doing.)  I recently switched to 4.2, and
>> in the process, modified the disk layout drastically (splitting what was
>> previously 2 filesystems into 4).  Since neither of the old filesystems
>> would have fit into the new slots, I just created symbolic links to the
>> parts I wanted to put on another filesystem, and it all worked!
>> 
>> E.g., put /usr/src on /w/src:
>> 	mount /usr and /w appropriately
>> 	mkdir /w/src
>> 	ln -s /w/src /usr/src
>> 	cd /usr
>> 	restore r
>> 		... wait a long time ... :-)
>> 
>> =S

Thank you for your enlightening response.  You have reaffirmed
my faith in humanity (or at least some sane segment of it).

The machine I had trouble with is an ONYX C8002M.  We also have
a MASSCOMP MC500.  The MASSCOMP's OS is a MASSCOMP-engineered
hybrid of System III and BSD.  After reading your article I wanted
to see if our restor on the MASSCOMP was the BSD restore.  I
created a filesystem on a floppy disk (1280 blocks) and put one
small (3 blocks) file on it.  I unmounted the filesystem and
fsck'd it -- no problems.  I dumped it.  I re-mkfs'd it.  I restor'd
it.  I fsck'd it -- BAD FREE INODE COUNT IN SUPERBLOCK!!!!! 
How could such a simple operation cause such an error??  Not that
fsck couldn't fix the error, but there really shouldn't be any
errors at all!  Restor is one sick program!

Many, many thanks to Tom Teixeira of MASSCOMP for his article
on restor.

>> From: rdin!philabs!seismo!harpo!decvax!genrad!grkermit!masscomp!kobold!tjt
>>
>> Rest assured: it IS possible to restore a filesystem from incremental
>> dumps, although the manual is not explicit about various caveats that
>> must be observed.  Quoting from the 4.1BSD manual (I don't have a V7
>> manual handy, but I believe the V7, 4.1BSD and the System III manual
>> entries for restor(8) are fundamentally the same):
>> 
>>     The r option should only be used to restore a complete dump tape
>>     onto a clear file system or to restore an incremental dump tape
>>     onto this.  Thus
>> 
>> 	/etc/mkfs /dev/rrp0g 145673
>> 	restor r /dev/rrp0g
>> 
>>     is a typical sequence to restor a complete dump.  Another restor
>>     can be done to get an increment dump in on top of this.
>> 
>> What is not explicit here is that you have to read in the incremental
>> dump *immediately* after the full dump.  As Robert Perlberg
>> (rdin2!perl) points out, dump/restor work by dumping and restoring
>> files as inodes (i.e. identified by their inode number, not their file
>> names).  Restoring an incremental dump tape will not overwrite an
>> existing inode unless that inode was modified between the base dump and
>> the incremental dump.  However, if you create some files (inodes) between
>> reading the base tape and the incremental tape then one of these newly
>> created files could easily use the same inode as some file that was
>> created between the time of the base dump and the time of the
>> incremental dump.

What Tom isn't considering here is that a lot of mixing up of
the filesystem can take place between dumps which would result
in reassigning of inodes.

>> The manual also makes a reference to running "icheck -s" between tapes
>> of a multivolume dump if restor is interrupted.  The icheck program is
>> considerably less thorough than fsck and only performs what fsck calls
>> phases 1 (checking blocks in inodes), 1b (rescanning for duplicate
>> blocks) 5 (check free list) and 6 (rebuild free list).  The command
>> "icheck -s" is used to rebuild the free list, which is required since
>> restor allocates blocks for files from the file system free list but
>> doesn't update the super block until the entire restore is completed,
>> hence the requirement for running "icheck -s" IF the restor was
>> interrupted.
>> 
>> Curiously enough, icheck is *not* part of system III since for most
>> purposes, fsck is better.  In the absence of icheck (or an improved
>> fsck) you would have to answer no to all entreaties during phases 2
>> (check pathnames), 3 (check connectivity) or 4 (check reference counts)
>> to remove, reattach, clear or otherwise tamper with your inodes.
>> Of course, you should get no bad or duplicate blocks in phase 1, and
>> you will probably get a bad free list in phase 5 and should rebuild the
>> free list.

What a headache!!!  As neat as that explaination is, why didn't
they just write a dump/restor system that would just work right?!?

>> As to *why* restor works this way:
>> 
>> The list of inodes is a nice, contiguous array on the disk while
>> dumping files by name requires walking the directory tree.  If you
>> prefer, it is fairly easy to use find and cpio to perform incremental
>> dumps by name.  The advantage of dump/restor is that a standalone
>> version of restor is available in case you clobber your root file
>> system.   There is no inherent reason for not having a standalone
>> version of cpio, but I'm not aware of any.  Going through the file
>> system to do your dumps (e.g. using cpio) will have the side effect of
>> changing the "last accessed" time of the file.  Changing the "last
>> accessed" time back using the -a option of cpio has the possibly worse
>> side effect of changing the "last changed" time of the file.

Terrific!!  Restor sucks, so use cpio.  Of course, cpio has this,
that, and the other things wrong with it, too, so we're right
back where we started.

>> In the past, various people have noted other problems with doing dumps
>> of an active file system which are consequences of dump avoiding the
>> normal file system mechanisms of opening and reading files.  In
>> particular, when unix's in-core inode is inconsistent with the inode on
>> the disk, it is possible to interpret ordinary file blocks as indirect
>> blocks (consider what happens if a file is deleted and one of its
>> former indirect blocks gets reallocated as an ordinary file block, and
>> the new file block gets written to the disk *before* the inode (now
>> empty) gets updated).  Since this is also a potential problem if the
>> system crashes, it should be fixed by modifying the system to
>> synchronously write deleted inodes to the disk before adding the
>> indirect blocks to the free list.  I don't recall offhand whether the
>> changes made in 4BSD to improve the file system reliability included
>> this change or not.
>> -- 
>> 	Tom Teixeira,  Massachusetts Computer Corporation.  Westford MA
>> 	...!{ihnp4,harpo,decvax}!masscomp!tjt   (617) 692-6200 x275

Doesn't really matter, does it, since Berkeley dump does use
the normal file system mechanisms.

Well, thanks again for reasuring me that I am not the only one
who has had trouble with this.  I am surprised, though, that
there is so little momentum toward the elimination of this problem.
Berkeley has proven that it can be done.  How much more data
needs to be lost before the "Controllers of the Source" give
us "binary babies" a more reasonable way to safeguard our systems?

Robert Perlberg
Resource Dynamics Inc.
New York
philabs!rdin!rdin2!perl