News archiver for 3b1 tape drives, Part 1 of 2
Vernon C. Hoxie
vern at zebra.UUCP
Sat Jan 6 22:35:35 AEST 1990
This collection of script and 'c' programs permit the archiving
of news groups to 3b1 tape drives and permit retrieval by subject.
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of archive 1 (of 2)."
# Contents: Digest Install MANIFEST Makefile README conv.prep.sh
# conv.rd.sh conv.sq.sh convert.sh copy.sh group.c header.sh prep.sh
# purge.sh rd.tape.sh read.vtoc.c save.list squirrel.sh strt.hdr.sh
# Wrapped by root at quagga on Fri Jan 5 21:24:46 1990
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f Digest -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"Digest\"
else
echo shar: Extracting \"Digest\" \(4946 characters\)
sed "s/^X//" >Digest <<'END_OF_Digest'
X This is a collection of shell scripts and 'c' programs designed
Xto aid in archiving news articles. The format permits these articles to be
Xrecovered individually according to the 'Subject:' line of each article.
X
X The collection consists of the following programs:
X
X'squirrel.sh': This links the articles in the news directories selected
Xby the user edited 'save.list' file. It is to be run by 'cron' and
Xsends reports to the news administrator as to the status of the
Xcollection. A support file containing the 'Message-Id:' of each article
Xis maintained to preclude duplication of articles due to cross postings.
X
X'prep.sh': This performs the preliminary processing prior to writing to
Xtape. A 'contents' file is generated from the 'Subject:' line of each
Xarticle and then 'contents' is sorted and regrouped according to subject
Xso the follow-up titles appear together in 'contents'. The articles are
Xthen compressed and moved from the 'arcdir' to 'savdir'.
X
XBASIC SCRIPTS:
X'wr.tape.sh': This performs the actual write operation after
Xdetermining that there is room on the tape for the collection of
Xarticles. If the collection is larger than the available space, the
Xcollection is truncated. Articles successfully transferred to tape are
Xpurged from the 'arcdir'.
X
X'rd.tape.sh': This reads the volume table of contents from the tape and
Xpermits the user to select one chapter from the tape for review. The
X'contents' file of the selected chapter is then presented to the user so
Xthat individual articles can be selected. The desired articles are then
Xcopied out into a subdirectory of the '/news/.tape.ops' directory for
Xretrieval by the user.
X
X'copy.sh': This is included as an aid in recovering from a verification
Xerror during a write operation. It copies existing chapters on a bad
Xtape to another tape so that additions chapters can be added to the
Xcollection.
X
X'header.sh': The definitions for the various file names and directories
Xare listed here for use by all the shell scripts so that these names are
Xconsistent for the complete set.
X
XSUPPORT 'C' PROGRAMS:
X'read.vtoc.c': This reads the '/tmp/vtoc' file generated by 'Tgetname -b'
Xcall. It is used by the 'rd.tape' and 'wr.tape' scripts for information
Xneeded for accessing the tape. See the comments in 'read.vtoc.c' for
Xmore information.
X
X'group.c': This performs the grouping of similar subject titles in
Xthe 'contents' file before writing to tape.
X
X'pick.files.c': This is the display and selection program for retrieving
Xthe desired articles.
X
XREFERENCE FILES:
X'save.list': This is the user defined list of news directories which
Xare recursively searched for articles to be "squirreled" for later
Xtransfer to tape.
X
X'params': This is a generated file which is maintained by the various
Xprograms to transfer data from one operation to the next.
X
X'link.list': Another generated file which contains the 'Message-Id:'
Xlines from the articles which have 'squirreled'. These Id's are kept on
Xfile for two months at which time the list is purged by 'squirrel.sh'.
X
XCONVERT SCRIPTS:
X An addition set of scripts are included with this distribution
Xto convert existing tapes of the '/news/...' articles to the format used
Xby the 'rd.tape.sh'. These are run from '/news/.start' to prevent
Xinterference with normal data collection processes.
X
X'strt.hdr.sh': The definitions for the various file names and directories
Xare listed here for use by all the conversion scripts so that these file
Xnames are used consistently for all the operations.
X
X'convert.sh': The state machine which controls the separate operations
Xto convert regular tapes to this format.
X
X'conv.rd': This consecutively reads the chapters from the AT&T type
Xtapes into '/news/.start/recover/news/*..' so that they can be processed
Xby 'conv.sq' until sufficient articles are available to fill a chapter
Xin the compressed format.
X
X'conv.prep': A copy of 'prep.sh' which accesses the special directories
Xused by the conversion process.
X
X'conv.wr': A copy of 'wr.tape' for use by the conversion process.
X
X'rd.params': A generated file which controls the reading of the tapes.
X
X'wr.params': A generated file which controls the writing of the
Xcompressed files to tape. It is identical to the 'params' file used by
Xthe regular process but is maintained separately in order to preclude
Xinterference between the processes.
X
XADDITIONAL SCRIPT:
X'purge.sh': This is not necessary for the operation of any of the tape
Xoperations. It searches the entire '/news/..' file systems for empty
Xdirectories and deletes these derelicts generated by cross postings.
XI run it once a week.
X
XINSTALLATION AIDS:
X'install.sh': This shell script makes the necessary directories and
Xtransfers the programs to the places they will be found by 'header.sh'
Xand 'strt.hdr.sh'. It also calls the 'Makefile' to produce the 'c' code
Xbinaries. The configuration produced is that which is to be used by
Xnormal operations.
X
END_OF_Digest
if test 4946 -ne `wc -c <Digest`; then
echo shar: \"Digest\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f Install -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"Install\"
else
echo shar: Extracting \"Install\" \(2194 characters\)
sed "s/^X//" >Install <<'END_OF_Install'
XBASE=/news/.tape.ops
XGRP=news
XUSER=netnews
X
Xif [ ! -d $BASE ]
Xthen
X mkdir $BASE
X chgrp $GRP $BASE
X chown $USER $BASE
Xfi
X
Xfor i in header.sh squirrel.sh prep.sh wr.tape.sh rd.tape.sh
Xdo
X echo "Moving '$i' to '$BASE'"
X cp "$i" $BASE
Xdone
Xchmod 755 $BASE/*
Xecho "Moving 'save.list' to '$BASE'"
Xcp save.list $BASE
Xchgrp $GRP $BASE/*
Xchown $USER $BASE/*
X
Xif [ ! -d $BASE/docs ]
Xthen
X mkdir $BASE/docs
X chgrp $GRP $BASE/docs
X chown $USER $BASE/docs
Xfi
X
Xfor i in README Digest Theory
Xdo
X echo "Moving '$i' to '$BASE/docs'"
X cp "$i" $BASE/docs
Xdone
Xchgrp $GRP $BASE/docs/*
Xchown $USER $BASE/docs/*
X
Xif [ ! -d $BASE/src ]
Xthen
X mkdir $BASE/src
X chgrp $GRP $BASE/src
X chown $USER $BASE/src
Xfi
X
Xfor i in read.vtoc.c pick.files.c group.c Makefile
Xdo
X echo "Moving '$i' to '$BASE/src'"
X cp "$i" $BASE/src
Xdone
X
Xhere=$PWD
Xcd $BASE/src
Xmake
Xchgrp $GRP $BASE/src/*
Xchown $USER $BASE/src/*
Xcd $here
X
Xline="20 02 * * * /bin/su netnews % ksh -c /usr/lib/news/tape.ops/squirrel"
Xecho "Do you want me to add:\n$line\nto '/usr/lib/crontab'? <y/n> \c"
Xread ch
Xif [ "$ch" = "y" ]
Xthen
X cp /usr/lib/crontab /tmp/cron
X echo "$line" >> /tmp/cron
X if [ $? = 0 ]
X then
X mv /tmp/cron /usr/lib/crontab
X fi
Xfi
X
Xecho "Do you want to run 'purge'? <y/n> \c"
Xread ch
Xif [ "$ch" = "y" ]
Xthen
X cp purge.sh /usr/local/bin
X chmod 0755 /usr/local/bin/purge.sh
X chgrp bin /usr/local/bin/purge.sh
X chown bin /usr/local/bin/purge.sh
X line="\
X10 03 * * 3 /bin/su netnews % ksh -c /usr/local/bin/purge > /dev/null"
X echo "Do you want me to add:\n$line\nto '/usr/lib/crontab'? <y/n> \c"
X read ch
X if [ "$ch" = "y" ]
X then
X cp /usr/lib/crontab /tmp/cron
X echo "$line" >> /tmp/cron
X if [ $? = 0 ]
X then
X mv /tmp/cron /usr/lib/crontab
X fi
X fi
Xfi
X
XSTART=/news/.start
Xecho "Do you install the conversion scripts in '$START'? <y/n> \c"
Xread ch
Xif [ "$ch" = "y" ]
Xthen
X mkdir $START
X chgrp $GRP $START
X chown $USER $START
X for i in strt.hdr.sh convert.sh conv.rd.sh conv.sq.sh \
Xconv.prep.sh wr.tape.sh
X do
X echo "Moving '$i' to '$START'"
X cp $i $START
X done
X mv $START/wr.tape.sh $START/conv.wr.sh
X chmod 0755 $START/*
X echo "Moving 'save.list' to '$START'"
X chgrp $GRP $START/*
X chown $USER $START/*
Xfi
X
Xecho "Installation complete."
END_OF_Install
if test 2194 -ne `wc -c <Install`; then
echo shar: \"Install\" unpacked with wrong size!
fi
chmod +x Install
# end of overwriting check
fi
if test -f MANIFEST -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"MANIFEST\"
else
echo shar: Extracting \"MANIFEST\" \(774 characters\)
sed "s/^X//" >MANIFEST <<'END_OF_MANIFEST'
X File Name Archive # Description
X-----------------------------------------------------------
X Digest 1
X Install 1
X MANIFEST 1 This shipping list
X Makefile 1
X README 1
X Theory 2
X conv.prep.sh 1
X conv.rd.sh 1
X conv.sq.sh 1
X convert.sh 1
X copy.sh 1
X group.c 1
X header.sh 1
X pick.files.c 2
X prep.sh 1
X purge.sh 1
X rd.tape.sh 1
X read.vtoc.c 1
X save.list 1
X squirrel.sh 1
X strt.hdr.sh 1
X wr.tape.sh 2
END_OF_MANIFEST
if test 774 -ne `wc -c <MANIFEST`; then
echo shar: \"MANIFEST\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f Makefile -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"Makefile\"
else
echo shar: Extracting \"Makefile\" \(254 characters\)
sed "s/^X//" >Makefile <<'END_OF_Makefile'
XCFLAGS=-O -v
XLDFLAGS=-s
X
XSHAREDLIB=/lib/crt0s.o /lib/shlib.ifile
X
XTARGETS=group pick.files read.vtoc
X
XOBJS=$(TARGETS).o
X
Xall: $(TARGETS)
X
X$(TARGETS): $$@.o
X $(LD) $(LDFLAGS) $(SHAREDLIB) $? $(LIBS) -o $@
X chgrp news $@
X chown netnews $@
X
Xclean:
X rm *.o;
END_OF_Makefile
if test 254 -ne `wc -c <Makefile`; then
echo shar: \"Makefile\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f README -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"README\"
else
echo shar: Extracting \"README\" \(2687 characters\)
sed "s/^X//" >README <<'END_OF_README'
X This collection of programs was designed to permit the archiving
Xof news article directly to UNIX PC type tapes. Articles are protected
Xfrom 'expire' by linking them to a separate directory. Because some
Xusers have the news written to separate partition of the disk or to a
Xsecond disk, all operations and the linking process is accomplished in a
Xsub directory of /news. A dot file is used to avoid possible conflicts
Xwith the regular news programs.
X
X News groups to be archived are selected by entering their names
Xin full path name format in '/news/.tape.ops/save.list'. Note that the
Xsearch algorithm is recursive and any directories encountered are
Xsearched in turn. This means that an entry of '/news/unix-pc'
Xin 'save.list' will search all the groups included under 'unix-pc' such
Xas '/news/unix-pc/general', '/news/unix-pc/sources, etc. In some cases
Xthis may result in archiving of groups not really wanted.
X
X The 'install.sh' script makes all the needed directories and
Xtransfers the scripts to the locations the other scripts expects to
Xfind them. It also runs 'make' to produce the binaries for the operations
Xwhich were not readily implemented in shell. It should be noted that
Xthese scripts must run by '/bin/ksh' as the bourne shell does not
Xdo 'functions' or '(( .. ))' type calculations.
X
X 'Squirrel.sh' should be run by cron at least as often as the
Xretention duration of 'expire' so that articles are not deleted before
Xthey are 'squirreled'. There is a line in 'install.sh' which can be
Xuncommented which will write an appropriate entry into your 'crontab'.
X
X Another set of scripts are included to convert tapes which were
Xwritten by the regular processes. They are designed to be run from
X'/news/.start' to keep them from interfering with the regular 'run'
Xoperations. After all existing tapes have been converted, the entire
X'/news/.start' directory may be deleted. These are set up as an option
Xin 'install.sh'.
X
X Although not really part of the archiving procedures, 'purge' is
Xincluded to delete empty directories from the /news heading. These
Xdirectories are remnants of cross postings long since forgotten. This
Xtoo is an option in 'install.sh'.
X
X Hopefully, the documentation included in "Theory" and "Digest"
Xwill provide a guide into using and modifying these scripts. Comments
Xin the scripts and in the '.c' files should also be referenced for more
Xinformation.
X
X If you run into trouble, want additional information or have
Xsuggestions, please e-mail at the following address.
X
XVernon C. Hoxie {ncar,nbires,boulder,isis}!scicom!zebra!vern
X3975 W. 29th Ave. voice: 303-477-1780
XDenver, Colo., 80212 TB+ uucp: 303-455-2670
END_OF_README
if test 2687 -ne `wc -c <README`; then
echo shar: \"README\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f conv.prep.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"conv.prep.sh\"
else
echo shar: Extracting \"conv.prep.sh\" \(2555 characters\)
sed "s/^X//" >conv.prep.sh <<'END_OF_conv.prep.sh'
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.start/strt.hdr.sh # Use this for start up
Xfi
X
Xecho `date +%T`" Starting prep routines for transfer to tape."
X
X# Clean out any previous effort to generate a 'savdir' etc. Then
X# create new ones.
X
Xif [ -d $SAVDIR ]
Xthen
X rm -r $SAVDIR
Xfi
Xmkdir $SAVDIR
X
Xif [ -f $TMP1 ]
Xthen
X rm $TMP1
Xfi
Xif [ -f $TMP2 ]
Xthen
X rm $TMP2
Xfi
X
Xif [ -s $PARAMS ]
Xthen
X fnum=`awk - '{ if ( $0 ~ /Last fnum:/ ) print $3 }' $PARAMS`
Xelse
X echo "We are in trouble!"
X echo "There is no $PARAMS file so the 'squirrel'ed data is"
X echo "defective. Clean it up before proceeding."
X read
X exit 1
Xfi
X
X> $CONFILE
Xls -1 $ARCDIR > $PNTRS
X
Xecho `date +%T`" Reading 'Subject:' lines from 'arcdir'. "
X
X# Go through all the files in 'arcdir' and read the 'Subject: ' line.
X# Add a tab and the file number to the line and put it in 'contents'.
Xwhile
X read pntr
Xdo
X subj=`sed -n '/^Subject:/ {
X s/Subject: //p
X q
X }' $ARCDIR/$pntr`
X if [ -n "$subj" ]
X then
X echo "$subj""\t"$pntr >> $CONFILE
X fi
Xdone <$PNTRS
X
Xecho `date +%T`" Changing arrangement of the 'contents' file. "
X# Sort the 'contents' then run 'group' to place followup articles
X# following the original. 'Group' also produces 'c.pntrs' which is
X# a listing of the files in the same order that they appear
X# in the 'contents'.
X
Xsort -fd -o $CONFILE $CONFILE
X$SRCDIR/group
X
Xecho "\tYou should have a destination tape installed in the drive at
X\tthis point. It will then have completed the retension phase
X\tbefore you need it."
X
Xecho `date +%T`" Moving and compressing files to be taped. "
X# Get the 'arcdir' file names from 'contents' and transfer to 'savdir'
X# while compressing. Be careful not to use up the last 5 percent of
X# disk. Strip the "Path:" line from the heading in the process.
X
Xwhile
X read pntr
Xdo
X sed '/^Path:/d' $ARCDIR/$pntr | compress > $SAVDIR/$pntr.Z
X chgrp $GRP $SAVDIR/$pntr.Z
X chown $OWN $SAVDIR/$pntr.Z
X set `df $NEWSDEV`
X if [ $3 -lt $PERCENT ]
X then
X total=`cat $PNTRS | wc -l`
X lnum=`fgrep -nx $pntr $PNTRS | cut -f1 -d":"`
X let needed="$total-$lnum"
Xecho "You are running low on disk space. There are only $3 blocks
Xremaining. So far, $lnum of $total files have been compressed.
X A. Single step through the rest of the list.
X B. Open another window and transfer data to floppy disks.
X C. Go ahead and put what we have on tape, save the rest.
X D. Quit for now.
XEnter A, B, C, or D. <cr>"
X read ans
X if [ "$ans" = "d" -o "$ans" = "D" ]
X then
X cd $currdir
X exit 1
X elif [ "$ans" = "c" -o "$ans" = "C" ]
X then
X break
X fi
X fi
Xdone < $CPNTRS
X
Xexit 0
END_OF_conv.prep.sh
if test 2555 -ne `wc -c <conv.prep.sh`; then
echo shar: \"conv.prep.sh\" unpacked with wrong size!
fi
chmod +x conv.prep.sh
# end of overwriting check
fi
if test -f conv.rd.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"conv.rd.sh\"
else
echo shar: Extracting \"conv.rd.sh\" \(3258 characters\)
sed "s/^X//" >conv.rd.sh <<'END_OF_conv.rd.sh'
X# This is the sequence of operations to read articles from a tape
X# prepared using 'prep' and 'wr.tape'.
X
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.start/strt.hdr.sh
Xfi
X
X
X# This should be the start of the program
X
Xecho `date +%T`" Starting read operations. "
Xtoday=`date '+%y-%m-%d'`
Xif [ ! -f $RDPARMS ]
Xthen
X > $RDPARMS
X chgrp $GRP $RDPARMS
X chown $OWN $RDPARMS
Xfi
X
X# Check that the 'params' file has the proper entries.
Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) ok1 = 1
X if ( $0 ~ /Date last taped:/ ) ok2 = 1
X if ( $0 ~ /Next chapter number:/ ) ok5 = 1
X }
X END{
X if ( ok1 != 1 ) print "Name of work tape: "
X if ( ok2 != 1 ) print "Date last taped: 0"
X if ( ok5 != 1 ) print "Next chapter number: 0"
X }' $RDPARMS >> $RDPARMS
X
Xret=1
Xch=x
Xwhile [ $ret != 0 ]
Xdo
X Tgetname -t
X ret=$?
X if [ $ret != 0 ]
X then
Xecho "Please insert a source tape. <cr> to continue, Q)uit. \c"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit 1
X fi
X ret=1
X else
X# From here on, the tape drive is locked, any 'exit' must perform
X# 'Tgetname -l' to unlock the drive befor leaving.
X Tgetname +l
X Tgetname -v > /dev/null
X set `Tgetname -r | tr -d "'"`
X name=$1
X $SRCDIR/read.vtoc -w9 > this.tape
X if [ -f old.tape ]
X then
X diff this.tape old.tape > /dev/null
X ret=$?
X if [ $ret != 0 ]
X then
Xecho "This doesn't seen to be the tape that I have been reading.
XWaiting for another tape to be inserted.
X<cr> to continue. Q)uit \c"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X Tgetname -l
X exit 1
X fi
X fi
X else
Xecho "There isn't any info about any previous tape. If this is the beginning
Xof a tape to be converted, <cr> to continue. Q)uit \c"
X Tgetname -l
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit 1
X fi
Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) $2 = "'"$name"'"
X if ( $0 ~ /Date last taped:/ ) $2 = "'"$today"'"
X if ( $0 ~ /Next chapter number:/ ) $2 = "1"
X print $1" "$2 >> "'$TMP1'"
X }' $RDPARMS
X mv $TMP1 $RDPARMS
X ret=0
X fi
X fi
Xdone
X
X# Get params
Xtdate=`sed -n 's/Date last taped:\ //p' $RDPARMS`
Xchap=`sed -n 's/Next chapter number:\ //p' $RDPARMS`
X
Xset `$SRCDIR/read.vtoc -r$chap`
Xoffset=$2
Xblocks=$3
Xcat this.tape
Xbase=$PWD
Xif [ -d recover ]
Xthen
X echo `date +%T`" Purging old 'recover' directory. "
X rm -r recover
Xfi
Xmkdir recover
Xcd recover
X
Xecho `date +%T`" Reading chapter $chap from tape '$name' starting \
Xat $offset offset."
Xdbuf -iT248O$offset /dev/rft3 2>/dev/nul | \
X execStrip tapecpio -icdR"T248" >/dev/null 2>$WRKDIR/errs
X
Xret=$?
Xif [ $ret != 0 ]
Xthen
X echo "ret = "$ret
X echo "Whoops. There was an error during the reading of '$name'.
XBailing out! Please rerun this program."
X Tgetname -l
X exit 1
Xfi
Xecho `date +%T`" Rewinding tape."
Xnohup Tgetname -v >/dev/null 2>/dev/null & # to rewind the tape
X
Xcd $base
X
X(( chap=chap+1 ))
X$SRCDIR/read.vtoc -r$chap > /dev/null
Xret=$?
Xif [ $chap -ge $ret ]
Xthen
X rm old.tape
X chap=1
X echo "This was the last chapter on this tape. Next time insert \
Xa different tape."
X Tgetname -l
Xelse
X mv this.tape old.tape
Xfi
Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) $2 = "'"$name"'"
X if ( $0 ~ /Next chapter number:/ ) $2 = "'"$chap"'"
X print $1" "$2 >> "'$TMP1'"
X }' $RDPARMS
X mv $TMP1 $RDPARMS
Xrm -f /tmp/tape.vtoc
Xexit 0
END_OF_conv.rd.sh
if test 3258 -ne `wc -c <conv.rd.sh`; then
echo shar: \"conv.rd.sh\" unpacked with wrong size!
fi
chmod +x conv.rd.sh
# end of overwriting check
fi
if test -f conv.sq.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"conv.sq.sh\"
else
echo shar: Extracting \"conv.sq.sh\" \(3938 characters\)
sed "s/^X//" >conv.sq.sh <<'END_OF_conv.sq.sh'
X# The purpose of this script is to 'squirrel' away files in the news
X# files for transfer to tape. By linking these files to entries in
X# the 'arcdir', they are protected from 'expire' yet do not use up an
X# great amount of disk space.
X
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.start/strt.hdr.sh
Xfi
X
Xecho `date +%T`" Running 'squirrel' to link files into 'arcdir'. "
X
X# To commence, let's find out the status of our files and generate
X# the directories and file entries which are missing.
X
Xif [ ! -d $ARCDIR ]
Xthen
X mkdir $ARCDIR
X chmod 755 $ARCDIR
Xfi
X
Xif [ ! -s $PARAMS ]
Xthen
X > $PARAMS
X chgrp $GRP $PARAMS
X chown $OWN $PARAMS
X fnum = 1
Xelse
X fnum=`sed -n 's/Next fnum to link:\ //p' $PARAMS`
Xfi
X
X# Check that the 'params' file has the proper entries.
Xawk -F" " - '{
X if ( $0 ~ /Next fnum to link:/ ) ok1 = 1
X if ( $0 ~ /Number of bytes held:/ ) ok2 = 1
X if ( $0 ~ /Date last squirreled:/ ) ok3 = 1
X if ( $0 ~ /Name of work tape:/ ) ok4 = 1
X if ( $0 ~ /Blocks of offset:/ ) ok5 = 1
X if ( $0 ~ /Date last taped:/ ) ok6 = 1
X if ( $0 ~ /Blocks available:/ ) ok7 = 1
X if ( $0 ~ /Next chapter number:/ ) ok8 = 1
X if ( $0 ~ /Compression factor:/ ) ok9 = 1
X }
X END{
X if ( ok1 != 1 ) print "Next fnum to link: 0"
X if ( ok2 != 1 ) print "Number of bytes held: 0"
X if ( ok3 != 1 ) print "Date last squirreled: 0"
X if ( ok4 != 1 ) print "Name of work tape: News-000"
X if ( ok5 != 1 ) print "Blocks of offset: 0"
X if ( ok6 != 1 ) print "Date last taped: 0"
X if ( ok7 != 1 ) print "Blocks available: 45566"
X if ( ok8 != 1 ) print "Next chapter number: 0"
X if ( ok9 != 1 ) print "Compression factor: 1000"
X }' $PARAMS >> $PARAMS
X
Xtoday=`date '+%y-%m-%d'`
X
X# Check for a link list and either create a new one or copy
X# the current one to wrk.links for upadating.
X
Xif [ -s $LINKS ]
Xthen
X cp $LINKS $KLINKS
Xelse
X > $KLINKS
Xfi
X
Xx=fnum
Xwhile [ -f $ARCDIR/$x ]
Xdo
X rm -f $ARCDIR/$x
X (( x = x + 1 ))
Xdone
X
Xrm -f $WRKDIR/contents
Xrm -f $GOOFS
X
Xfunction recur
X{
X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ]
X then
X return -1
X fi
X typeset this next
X this=$1
X set `ls -1 $1`
X while [ $1 ]
X do
X next=$this/$1
X if [ -d $next ]
X then
X recur $next
X fi
X if [ -f $next ]
X then
X id=`sed -n '/^Message-ID:/ {
X s/Message-ID: //p
X q
X }' $next | tr -d "/*?^$\[\]|"`
X if [ -n "$id" ]
X then
X if
X [ -z "`sed -n '/'"$id"'/p' $KLINKS`" ]
X then
X if ln $next $ARCDIR/$fnum
X then
X let fnum="fnum+1"
X echo "$id""\t"$today \
X >> $KLINKS
X else
X echo $next", "$ARCDIR/$fnum >> $GOOFS
X fi
X fi
X fi
X fi
X shift
X done
X}
X
Xwhile
X read fname
Xdo
X dname=`echo $fname | cut -f1 -d"#"`
X if [ -n "$dname" ]
X then
X echo `date +%T`" Squirreling "$dname.
X recur $WRKDIR/recover$dname
X fi
Xdone <$LIST
X
X# Fudge the sum by 80 bytes for each entry to account fot the 'contents'
X# file and the 'ss.filelist' which will added to the tape during write
X# operations.
Xnbytes=`ls -l $ARCDIR | awk - '{ sum = sum + $5 + 80 }
X END{ print sum }' - `
X
Xsort -d -o $LINKS $KLINKS
X
X> $TMP1
Xawk -F" " - '{ if ( $0 ~ /Next fnum to link:/ ) $2 = "'"$fnum"'"
X if ( $0 ~ /Number of bytes held:/ ) $2 = "'"$nbytes"'"
X if ( $0 ~ /Date last squirreled:/ ) $2 = "'"$today"'"
X print $1" "$2 >> "'$TMP1'"
X }' $PARAMS
Xmv $TMP1 $PARAMS
Xchgrp $GRP $PARAMS
Xchown $OWN $PARAMS
Xchgrp $GRP $LINKS
Xchown $OWN $LINKS
Xrm -f $KLINKS
X
Xoffset=`sed -n 's/Blocks of offset:\ //p' $PARAMS`
Xchap=`sed -n 's/Next chapter number:\ //p' $PARAMS`
Xfactor=`sed -n 's/Compression factor:\ //p' $PARAMS`
X
X(( min_size = ( MAXOFFSET - offset ) / (10 - chap) ))
X(( this = nbytes/$factor )) # blocks for this collection
X
Xecho `date +%T`" Squirrling completed.
X Recommended size of next tape entry: $min_size
X Approximate size of this collection: $this"
X
Xif [ -d recover ]
Xthen
X
X echo `date +%T`" Purging 'recover' directory."
X rm -r recover
Xfi
X
Xif [ "$chap" -lt 9 ]
Xthen
X (( this = this + 100 ))
Xfi
X
Xif [ "$this" -gt "$min_size" ]
Xthen
X exit 2
Xelse
X exit 0
Xfi
END_OF_conv.sq.sh
if test 3938 -ne `wc -c <conv.sq.sh`; then
echo shar: \"conv.sq.sh\" unpacked with wrong size!
fi
chmod +x conv.sq.sh
# end of overwriting check
fi
if test -f convert.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"convert.sh\"
else
echo shar: Extracting \"convert.sh\" \(1505 characters\)
sed "s/^X//" >convert.sh <<'END_OF_convert.sh'
X# This is the sequence of operations to read articles from a tape
X# prepared using 'prep' and 'wr.tape'.
X
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.start/strt.hdr.sh
Xfi
X
X# This is the start of the program
X
Xawk -F" " - '{ if ( $0 ~ /State of operations:/ ) ok1 = 1
X }
X END{
X if ( ok1 != 1 ) print "State of operations: 1"
X }' $RDPARMS >> $RDPARMS
X
Xif [ ! -s $PARAMS ]
Xthen
X echo "Next fnum to link: 1" > $PARAMS
X echo "Number of bytes held: 0" >> $PARAMS
X echo "Date last squirreled: 0" >> $PARAMS
X echo "Next chapter number: 1" >> $PARAMS
X echo "Blocks of offset: 0" >> $PARAMS
X chgrp $GRP $PARAMS
X chown $OWN $PARAMS
Xfi
X
Xstate=1
X
Xwhile [ 1 ]
Xdo
X state=`sed -n 's/State of operations:\ //p' $RDPARMS`
X case $state in
X 1) conv.rd.sh
X ret=$?
X if [ "$ret" = "0" ]
X then
X state=2
X else
X Tgetname -l
X exit
X fi;;
X 2) conv.sq.sh
X ret=$?
X case $ret in
X 0) state=1;;
X 1) tgetname -l
X exit;;
X 2) state=3;;
X esac;;
X 3) Tgetname -l
X conv.prep.sh
X ret=$?
X if [ "$ret" = 0 ]
X then
X state=4
X else
X exit
X fi;;
X 4) conv.wr.sh
X ret=$?
X if [ "$ret" = 0 ]
X then
X state=1
Xecho `date +%T`" Finished writing to new tape. Reinsert the source tape.
X<cr> to continue. Q)uit."
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit
X fi
X else
X exit
X fi;;
X esac
Xawk -F" " - '{ if ( $0 ~ /State of operations:/ ) $2 = "'"$state"'"
X print $1" "$2 >> "'$TMP1'"
X }' $RDPARMS
X mv $TMP1 $RDPARMS
X chgrp $GRP $RDPARMS
X chown $OWN $RDPARMS
X
Xdone
Xexit
END_OF_convert.sh
if test 1505 -ne `wc -c <convert.sh`; then
echo shar: \"convert.sh\" unpacked with wrong size!
fi
chmod +x convert.sh
# end of overwriting check
fi
if test -f copy.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"copy.sh\"
else
echo shar: Extracting \"copy.sh\" \(5516 characters\)
sed "s/^X//" >copy.sh <<'END_OF_copy.sh'
X# This script is presented as a recovery tool to be used if a verify
X# failure occurs when writing to tape using these scripts. It copies
X# data from one tape to another chapter by chapter.
X
X# It requires that the source tape be written by the 'wr.tape.sh' script
X# which is included in this collection of scripts. Changes might be made
X# to these scripts so that other data can be copied, but I haven't had
X# need to generate such revisions. That is left as an exercise for you.
X
X# A test is first run to determine that there is sufficient disk space
X# available to copy each chapter. If insufficient space is available,
X# the amount of additional space required is reported.
X
X# Space can be attained by copying some files to floppy disk. Also, if
X# you have the time to rerun the 'prep.sh' script, you can free disk space
X# by issuing the 'rm -r savdir' command..
X
X# These routines will overwrite existing tapes if you desire. This option
X# is made at the begining of the 'wr.tape.sh' operation. If you want to
X# add on to an existing tape, you must edit this script in the initialization
X# of the $PARAMS routine. You can also edit 'cp.params' from another window
X# while the read operation is running.
X
Xexport NEWSDEV=/dev/fp002
Xexport WRKDIR=/news/.tape.ops
Xexport CACHE=$WRKDIR/cache
Xexport ARCDIR=$WRKDIR/cache
Xexport PARAMS=$WRKDIR/cp.params
Xexport SAVDIR=$WRKDIR/cache
Xexport CONFILE=$WRKDIR/c.contents
Xexport SRCDIR=/news/.tape.ops/src
Xexport PNTRS=$WRKDIR/cp.pntrs
Xexport CPNTRS=$WRKDIR/cp.pntrs
Xexport TMP1=$WRKDIR/tmpp1
Xexport TMP2=$WRKDIR/tmpp2
Xexport COPY=1
Xexport GRP=news
Xexport OWN=netnews
Xexport PERCENT=6200 # This is the last five percent of disk space
Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes
X# If we add 2 blocks for header, B200 512 byte blocks
X# or 1DAA blocks per stream???
X
Xecho `date +%T`" Checking available memory and setting up files. "
X
XTgetname -t
Xret=$?
Xpau=0
Xwhile [ $pau -eq 0 ]
Xdo
X if [ $ret -ne 0 ]
X then
X echo \
X"Please insert a source tape. Q)uit or <cr> to continue. \c"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit 1
X fi
X Tgetname -t
X ret=$?
X else
X Tgetname -v > /dev/null
X biggest=`$SRCDIR/read.vtoc -g`
X set `df $NEWSDEV`
X avail=$3
X (( dif = biggest + $PERCENT - avail ))
X if [ $dif -gt 0 ]
X then
X echo "The copy operation requires $dif more blocks of disk space."
X exit
X fi
X pau=1
X fi
Xdone
X
Xif [ ! -s $PARAMS ]
Xthen
X > $PARAMS
Xfi
X
X# Check that the 'cp.params' file has the proper entries.
Xawk -F" " - '{ if ( $0 ~ /Number of bytes held:/ ) ok1 = 1
X if ( $0 ~ /Next fnum to link:/ ) ok2 = 1
X if ( $0 ~ /Name of work tape:/ ) ok3 = 1
X if ( $0 ~ /Blocks of offset:/ ) ok4 = 1
X if ( $0 ~ /Date last taped:/ ) ok5 = 1
X if ( $0 ~ /Blocks available:/ ) ok6 = 1
X if ( $0 ~ /Next chapter number:/ ) ok7 = 1
X if ( $0 ~ /Compression factor:/ ) ok8 = 1
X if ( $0 ~ /Next source chapter:/ ) ok9 = 1
X }
X END{ if ( ok1 != 1 ) print "Number of bytes held: 0"
X if ( ok2 != 1 ) print "Next fnum to link: 0"
X if ( ok3 != 1 ) print "Name of work tape: blank"
X if ( ok4 != 1 ) print "Blocks of offset: 0"
X if ( ok5 != 1 ) print "Date last taped: 1-1-1"
X if ( ok6 != 1 ) print "Blocks available: 45566"
X if ( ok7 != 1 ) print "Next chapter number: 1"
X if ( ok8 != 1 ) print "Compression factor: 1024"
X if ( ok9 != 1 ) print "Next source chapter: 1"
X }' $PARAMS >> $PARAMS
X
Xfunction clean
X{
X rm -f $CONFILE
X rm -f $CPNTRS
X rm -r $CACHE
X rm -f $PARAMS
X rm -f $TMP1
X rm -f $TMP2
X rm -f $WRKDIR/errs
X}
X
Xpau=0
Xwhile [ "$pau" -eq 0 ]
Xdo
X ret=1
X while [ $ret != 0 ]
X do
X echo `date +%T`" Starting read operations. "
X today=`date '+%y-%m-%d'`
X
X Tgetname -t
X ret=$?
X if [ $ret != 0 ]
X then
Xecho "Please insert the source tape. <cr> to continue, Q)uit. \c"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit 1
X fi
X ret=1
X else
X# From here on, the tape drive is locked, any 'exit' must perform
X# 'Tgetname -l' to unlock the drive befor leaving.
X Tgetname +l
X Tgetname -v > /dev/null
X set `Tgetname -r | tr -d "'"`
X name=$1
X fi
X done
X
X# Get params
X chap=`sed -n 's/Next source chapter:\ //p' $PARAMS`
X
X set `$SRCDIR/read.vtoc -r$chap`
X offset=$2
X blocks=$3
X base=$PWD
X if [ -d $CACHE ]
X then
X echo `date +%T`" Purging old 'cache' directory. "
X rm -r $CACHE
X fi
X mkdir $CACHE
X cd $CACHE
X $SRCDIR/read.vtoc -w9
X echo `date +%T`" Reading chapter $chap from tape '$name' starting \
Xat $offset offset."
X dbuf -iT248O$offset /dev/rft3 2>/dev/nul | \
X execStrip tapecpio -icdR"T248" >/dev/null 2>$WRKDIR/errs
X
X ret=$?
X if [ $ret != 0 ]
X then
X echo "ret = "$ret
X echo "\tThere was an error during the reading of the \
Xsource tape.
X\tBailing out! Please rerun this program."
X Tgetname -l
X exit 1
X fi
X echo `date +%T`" Rewinding tape."
X Tgetname -v >/dev/null 2>/dev/null
X Tgetname -l
X cd $base
X
X (( chap = chap + 1 ))
X $SRCDIR/read.vtoc -r$chap > /dev/null
X ret=$?
X if [ $chap -ge $ret ]
X then
X pau=1
X echo "This was the last chapter on this tape."
X fi
X
X > $TMP1
X awk -F" " - '{ if ( $0 ~ /Next source chapter:/ ) $2 = "'"$chap"'"
X print $1" "$2 >> "'$TMP1'"
X }' $PARAMS
X mv $TMP1 $PARAMS
X
X cp $ARCDIR/contents $CONFILE
X sed 's/\-//; s/^d//; /Z$/ !d; s/.Z//' < $ARCDIR/ss.filelist > $CPNTRS
X echo "Insert the target tape now. <cr> to continue, Q)uit"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X clean
X exit
X fi
X $WRKDIR/wr.tape.sh
X echo "Insert the source tape now. <cr> to continue, Q)uit"
X read ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X clean
X exit
X fi
Xdone
Xclean
Xexit 0
END_OF_copy.sh
if test 5516 -ne `wc -c <copy.sh`; then
echo shar: \"copy.sh\" unpacked with wrong size!
fi
chmod +x copy.sh
# end of overwriting check
fi
if test -f group.c -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"group.c\"
else
echo shar: Extracting \"group.c\" \(3720 characters\)
sed "s/^X//" >group.c <<'END_OF_group.c'
X/* This is a 'c' version of a 'sed/awk' script I first used to group */
X/* similar titles together in the contents. It isn't as fast as some */
X/* other algorithm might be but it beats the devil out of the script. */
X/* That script took 20 minutes to process 800 titles!!! */
X
X/* The 'contents' is first read into the memory space allocated to */
X/* 'r1'. 'r1' is then scanned to count the number of lines. Next */
X/* 'r2' is allocated of sufficient size to hold 'lncnt' entries of */
X/* 'item' and 'r3' is allocated to hold 'lncnt' pointers to 'item'. */
X/* The 'items' in 'r2' are filled in by a second scan of 'r1'. */
X/* 'line' contains a pointer into 'r1' to the start of each entry in */
X/* the 'contents' listing and 'leng' is the number of characters in */
X/* the title. */
X/* The top 'item' is then used as pattern for scanning the remainder */
X/* of 'r1' for entries which contain the pattern or which are contained */
X/* in the pattern. When a match is found, the '*line' is written to */
X/* 'r4' and the 'leng' value in 'r2' is set to zero to indicate to */
X/* subsequent searchs that that entry has been "found". After each */
X/* pass, the 'r4' array is sorted to length and the **char's are */
X/* added to 'r3'. */
X/* After the scan sequence is completed, the data is read out from 'r1' */
X/* to the reopened 'contents' in the newly ordered sequence. */
X
X#include <fcntl.h>
X#include <malloc.h>
X#include <stdio.h>
X
X#define CFILE "contents"
X#define PFILE "c.pntrs"
X
Xtypedef struct { char *line;
X int leng;
X } item;
X
XFILE *fds1, *fds2;
Xmain()
X
X{
X int i, k, fd, lcnt, done;
X long size;
X char *c1, *c2, *c3, *c4;
X item *i1, *i2, *i3;
X char **l1;
X char *r1;
X item *r2;
X char **r3;
X item *r4[100];
X if ( access( CFILE, 04 ))
X {
X perror( "No \"contents\" file." );
X exit( 1);
X }
X fd = open("contents", O_RDONLY );
X size = lseek( fd, 0, 2 );
X lseek( fd, 0, 0 );
X r1 = malloc( size );
X read( fd, r1, size );
X close( fd );
X for( c1 = r1, lcnt = 0; c1 <= r1 + size ; c1++ )
X if ( *c1 == '\n' ) lcnt++;
X lcnt += 2;
X r2 = (item *)calloc( lcnt, sizeof( item ));
X r3 = (char **)calloc( lcnt, sizeof( char * ));
X for( c1 = r1, i1 = r2; c1 < r1 + size, *c1; i1++ )
X {
X i1->line = c1;
X c2 = c1;
X while ( *++c1 )
X {
X if ( *c1 == '\t' ) i1->leng = c1 - c2;
X else if ( *c1 == '\n' )
X {
X *c1++ = '\0';
X break;
X }
X }
X }
X i1->line = '\0';
X i1->leng = 0;
X l1 = r3;
X for ( i1 = r2; *i1->line; i1++ )
X {
X if ( ! i1->leng ) continue;
X i = 0;
X for ( i2 = i1; i2->line; i2++ )
X {
X if ( ! i2->leng ) continue;
X if ( i1->leng <= i2->leng )
X {
X k = i2->leng - i1->leng;
X c1 = i1->line;
X c2 = i2->line;
X }
X else
X {
X k = i1->leng - i2->leng;
X c1 = i2->line;
X c2 = i1->line;
X }
X for ( k++; k; c2++, k-- )
X {
X c3 = c1;
X c4 = c2;
X while ( *c3++ == *c4++ )
X if ( *c4 == '\t' )
X {
X r4[i++] = i2;
X break;
X }
X }
X }
X done = 0;
X /* Now reorder the strings to put the shortest first. */
X while ( ! done )
X {
X done = 1;
X for ( k = 0; k < i - 1; k++ )
X {
X if ( r4[k]->leng > r4[k+1]->leng )
X {
X i3 = r4[k];
X r4[k] = r4[k+1];
X r4[k+1] = i3;
X done = 0;
X }
X }
X }
X /* Now move the pointers to the output array */
X for ( k = 0; k < i; k++ )
X {
X *l1++ = r4[k]->line;
X r4[k]->leng = 0;
X /* leng = 0 says this item already queued for output */
X }
X }
X if (( fds1 = fopen( CFILE, "w" )) == 0 )
X {
X perror( "Opening contents file" );
X exit( -1 );
X }
X
X if (( fds2 = fopen( PFILE, "w" )) == 0 )
X {
X perror( "Opening pointers file" );
X exit( -1 );
X }
X for ( l1 = r3; *l1; l1++ )
X {
X fprintf( fds1, "%s\n", *l1 );
X c1 = *l1;
X while ( *c1++ != '\t' );
X fprintf( fds2, "%s\n", c1 );
X }
X}
END_OF_group.c
if test 3720 -ne `wc -c <group.c`; then
echo shar: \"group.c\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f header.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"header.sh\"
else
echo shar: Extracting \"header.sh\" \(1202 characters\)
sed "s/^X//" >header.sh <<'END_OF_header.sh'
X
X# The purpose of this script is to prepare the 'squirrel'ed files
X# for transfer to tape. This consists of building a 'contents'
X# file from the "Subject" line in the posted articles. The 'contents'
X# is sorted and similar titles are moved together for easier reference.
X# The articles are then compressed and moved to the 'savdir' where
X# they can copied to tape.
X
Xexport NEWSDEV=/dev/fp002
Xexport CMDDIR=/usr/lib/news/tape.ops
Xexport WRKDIR=/news/.tape.ops
Xexport SRCDIR=$WRKDIR/src
Xexport ARCDIR=$WRKDIR/arcdir
Xexport PARAMS=$CMDDIR/params
Xexport LIST=$CMDDIR/save.list # Directory list to be saved
Xexport LINKS=$CMDDIR/link.list
Xexport KLINKS=$CMDDIR/old.links
Xexport GOOFS=$CMDDIR/goofs
Xexport SAVDIR=$WRKDIR/savdir
Xexport CONDIR=$WRKDIR/condir
Xexport CONFILE=$WRKDIR/contents
Xexport PNTRS=$WRKDIR/pntrs
Xexport CPNTRS=$WRKDIR/c.pntrs
Xexport RPNTRS=$WRKDIR/r.pntrs
Xexport GRP=news
Xexport OWN=netnews
Xexport TMP=$WRKDIR/tmps
Xexport TMP1=$WRKDIR/tmpp1
Xexport TMP2=$WRKDIR/tmpp2
Xexport PERCENT=6200 # This is the last five percent of disk space
Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes
X# If we add 2 blocks for header, B200 512 byte blocks
X# or 1DAA blocks per stream???
END_OF_header.sh
if test 1202 -ne `wc -c <header.sh`; then
echo shar: \"header.sh\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f prep.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"prep.sh\"
else
echo shar: Extracting \"prep.sh\" \(2540 characters\)
sed "s/^X//" >prep.sh <<'END_OF_prep.sh'
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.tape.ops/header.sh
Xfi
X
Xecho `date +%T`" Starting prep routines for transfer to tape."
X
Xcurrdir=$PWD
Xcd $WRKDIR
X
X# Clean out any previous effort to generate a 'savdir' etc. Then
X# create new ones.
X
Xif [ -d $SAVDIR ]
Xthen
X rm -r $SAVDIR
Xfi
Xmkdir $SAVDIR
X
Xif [ -f $TMP1 ]
Xthen
X rm $TMP1
Xfi
Xif [ -f $TMP2 ]
Xthen
X rm $TMP2
Xfi
X
Xif [ -s $PARAMS ]
Xthen
X fnum=`awk - '{ if ( $0 ~ /Last fnum:/ ) print $3 }' $PARAMS`
Xelse
X echo "\t\tWe are in trouble!"
X echo "\tThere is no $PARAMS file so the 'squirrel'ed data is"
X echo "\tdefective. Clean it up before proceeding."
X read
X exit 1
Xfi
X
X> $CONFILE
Xls -1 $ARCDIR > $PNTRS
X
Xecho `date +%T`" Reading 'Subject:' lines from 'arcdir'. "
X
X# Go through all the files in 'arcdir' and read the 'Subject: ' line.
X# Add a tab and the file number to the line and put it in 'contents'.
Xwhile
X read pntr
Xdo
X subj=`sed -n '/^Subject:/ {
X s/Subject: //p
X q
X }' $ARCDIR/$pntr`
X if [ -n "$subj" ]
X then
X echo "$subj""\t"$pntr >> $CONFILE
X fi
Xdone <$PNTRS
X
Xecho `date +%T`" Changing arrangement of the 'contents' file. "
X
X# Sort the 'contents' then run 'group' to place followup articles
X# following the original. 'Group' also produces 'c.pntrs' which is
X# a listing of the files in the same order that they appear
X# in the 'contents'.
X
Xsort -fd -o $CONFILE $CONFILE
X$SRCDIR/group
X
Xecho "\nYou should have a destination tape installed in the drive at
Xthis point. It will then have completed the retension phase
Xbefore you need it.\n"
X
Xecho `date +%T`" Moving and compressing files to be taped. "
X# Get the 'arcdir' file names from 'contents' and transfer to 'savdir'
X# while compressing. Be careful not to use up the last 5 percent of
X# disk.
Xwhile
X read pntr
Xdo
X sed '/^Path:/d' $ARCDIR/$pntr | compress > $SAVDIR/$pntr.Z
X chgrp $GRP $SAVDIR/$pntr.Z
X chown $OWN $SAVDIR/$pntr.Z
X set `df $NEWSDEV`
X if [ $3 -lt $PERCENT ]
X then
X total=`cat $PNTRS | wc -l`
X lnum=`fgrep -nx $pntr $PNTRS | cut -f1 -d":"`
X let needed="$total-$lnum"
Xecho "You are running low on disk space. There are only $3 blocks
Xremaining. So far, $lnum of $total files have been compressed.
X A. Single step through the rest of the list.
X B. Open another window and transfer data to floppy disks.
X C. Go ahead and put what we have on tape, save the rest.
X D. Quit for now.
XEnter A, B, C, or D. <cr>"
X read ans
X if [ "$ans" = "d" -o "$ans" = "D" ]
X then
X cd $currdir
X exit 1
X elif [ "$ans" = "c" -o "$ans" = "C" ]
X then
X break
X fi
X fi
Xdone < $CPNTRS
X
X# Now do the tape operations.
Xwr.tape.sh
END_OF_prep.sh
if test 2540 -ne `wc -c <prep.sh`; then
echo shar: \"prep.sh\" unpacked with wrong size!
fi
chmod +x prep.sh
# end of overwriting check
fi
if test -f purge.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"purge.sh\"
else
echo shar: Extracting \"purge.sh\" \(824 characters\)
sed "s/^X//" >purge.sh <<'END_OF_purge.sh'
X# The purpose of this script is 'purge' the /news files of unused
X# directories.
X
XTMP=/tmp/purge
X
Xfunction recur
X{
X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ]
X then
X return -1
X fi
X typeset this next
X this=$1
X set `ls -1 $1`
X while [ $1 ]
X do
X next=$this/$1
X if [ -d $next ]
X then
X recur $next
X if [ $? -lt 0 ]
X then
X echo $next >> $TMP
X rmdir $next
X fi
X fi
X shift
X done
X}
X
X# This is the real start of the script, the functions had to come first.
X
X(echo "To: System Administrator"
Xecho "Subject: '/news' directories\n"
Xecho "\tThe following '/news' directories have been purged.") > $TMP
X
Xrecur /news
X
Xcat $TMP | mail root
Xrm $TMP
X
X# Maybe we shouldn't have zapped this directory, put it back.
Xif [ ! -d "/news/.rnews" ]
Xthen
X mkdir /news/.rnews
X chgrp news /news/.rnews
X chown netnews /news/.rnews
Xfi
Xexit
END_OF_purge.sh
if test 824 -ne `wc -c <purge.sh`; then
echo shar: \"purge.sh\" unpacked with wrong size!
fi
chmod +x purge.sh
# end of overwriting check
fi
if test -f rd.tape.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"rd.tape.sh\"
else
echo shar: Extracting \"rd.tape.sh\" \(2202 characters\)
sed "s/^X//" >rd.tape.sh <<'END_OF_rd.tape.sh'
X# This is the sequence of operations to read articles from a tape
X# prepared using 'prep' and 'wr.tape'.
X
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.tape.ops/header.sh
Xfi
X
X
X# This should be the start of the program
X
Xecho `date +%T`" Starting read operations. "
X
Xwhile [ "$ret" != 0 ]
Xdo
X Tgetname -t 2/dev/null
X ret=$?
X if [ "$ret" != 0 ]
X then
X echo "Please insert a tape. Q)uit or C)ontinue. \c"
X read -r ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X exit
X fi
X ret=1
X else
X# From here on, the tape drive is locked, any 'exit' must perform
X# 'Tgetname -l' to unlock the drive before leaving.
X Tgetname +l
X Tgetname -b > /dev/null 2>/dev/null
X ret=$?
X set `Tgetname -r | tr -d "'"`
X name=$1
X offset=$2
X if [ $offset = 0 ]
X then
X echo "\nYou may have inserted the wrong tape.
XThis tape has no data written to it. Enter \"g\" when you have
Xanother tape inserted. Enter \"q\" to quit."
X read -r ch
X if [ "$ch" = "q" -o "$ch" = "Q" ]
X then
X Tgetname -l
X exit
X else
Xecho "\nWaiting for another tape to be inserted.
XPress any key to continue. Q)uit \c"
X read -r ch
X if [ "ch" = "q" -o "$ch" = "Q" ]
X then
X Tgetname -l
X exit
X fi
X fi
X ret=1
X fi
X fi
Xdone
X
X# 'read.vtoc -w9' will write all used chapters to standard out in column
X# format.
X
Xecho `date +%T`" The contents of tape \""$name"\" are:"
X $SRCDIR/read.vtoc -w9
Xecho "Which chapter do you wish to access? \c"
Xread chap
X
Xset `$SRCDIR/read.vtoc -r$chap`
Xoffset=$2
Xblocks=$3
X
Xbase=$PWD
Xif [ ! -d $name ]
Xthen
X mkdir $name
Xfi
Xcd $name
Xif [ ! -d $chap ]
Xthen
X mkdir $chap
Xfi
Xcd $chap
X
Xtapecpio -icuxRT32O$offset contents < /dev/rft3 2>/dev/null
Xret=$?
Xif [ $ret != 0 ]
Xthen
X echo "ret = "$ret
X echo "\tThere was an error during the reading of the source tape.
X\tBailing out! Please rerun this program."
X Tgetname -l
X exit 1
Xfi
X$SRCDIR/pick.files
Xecho `date +%T`" Searching for files."
Xlist=`cat r.pntrs`
Xdbuf -iT124O$offset /dev/rft3 | \
X execStrip tapecpio -icvR"T124" $list 1> /dev/null 2>$WRKDIR/errs
X
Xecho `date +%T`" Rewinding tape."
XTgetname -v # to rewind the tape
XTgetname -l
Xecho `date +%T`" Your recovered files are in:\n"$PWD
X
Xrm -f /tmp/tape.vtoc
Xrm -f r.pntrs
Xrm -f ss.filelist
X
Xcd $base
Xexit
END_OF_rd.tape.sh
if test 2202 -ne `wc -c <rd.tape.sh`; then
echo shar: \"rd.tape.sh\" unpacked with wrong size!
fi
chmod +x rd.tape.sh
# end of overwriting check
fi
if test -f read.vtoc.c -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"read.vtoc.c\"
else
echo shar: Extracting \"read.vtoc.c\" \(3698 characters\)
sed "s/^X//" >read.vtoc.c <<'END_OF_read.vtoc.c'
X/* This is one way of getting around that damned 'TdspSSss' program */
X/* makes a window that you can't kill. */
X
X/* Flags: */
X/* c print [num] chapter number and offset, for writing */
X/* d debug, prints data about all 9 chapters */
X/* g return the size of the larget chapter */
X/* w write, print data about all used chapters */
X/* r read, print chapter offset and size, only one */
X/* entry if a number is included. */
X/* - ignored */
X/* [0-9] select specific chapter data */
X/* if a chapter number is present with either 'c' or 'w' */
X/* data relevant to that entry number is written to */
X/* '/tmp/ss.osets'. */
X/* The return value is the chapter number of the next blank */
X/* chapter unless [num] is used. Then it is [num]. */
X
X#include <stdio.h>
X#include <fcntl.h>
X#include <string.h>
X
X#define VTOC "/tmp/tape.vtoc"
Xstruct names {
X char text[46];
X short files;
X int start;
X int bytes;
X};
Xint fd, cflag, dflag, gflag, wflag, rflag;
Xstruct names entry;
X
Xmain(argc, argv)
Xint argc;
Xchar *argv[];
X{
X int i, num;
X char *p, ch;
X i = 1;
X cflag = 0;
X dflag = 0;
X gflag = 0;
X rflag = 0;
X wflag = 0;
X num = -1;
X while ( i < argc )
X {
X p = argv[i];
X while ( ch = *p++ ) switch( ch )
X {
X case '0':
X case '1':
X case '2':
X case '3':
X case '4':
X case '5':
X case '6':
X case '7':
X case '8':
X case '9':num = ch & 0x0f; break;
X case 'c':cflag = 1; break;
X case 'd':dflag = 1; break;
X case 'g':gflag = 1; break;
X case 'r':rflag = 1; break;
X case 'w':wflag = 1; break;
X case '-':
X default: break;
X }
X i++;
X }
X if (( fd = open( VTOC, O_RDONLY)) == -1 )
X {
X perror("Can't open /tmp/tape.vtoc");
X exit( -1 );
X }
X lseek( fd, 520, 0 );
X if ( wflag | cflag ) i = w_info( num );
X else if ( rflag ) i = r_info( num );
X else if ( gflag ) i = greatest();
X close( fd );
X exit( i );
X}
X
Xint w_info( num )
Xint num;
X{
X int fd2, i, j, k, m, n;
X char *p;
X char buf1[20];
X char name[10];
X char id[16];
X char comment[26];
X if ( num != '\0' ) j = num;
X else j = -1;
X k = 0;
X if ( wflag ) printf(
X"Chap Name Comment Files Offset Size\n");
X for (i=1; i <= 9; i++)
X {
X read( fd, &entry, sizeof(entry) );
X if ( entry.text[0] < '\0' )
X {
X if ( dflag ){
X entry.text[0] = '\0';
X entry.text[8] = '\0';
X entry.text[21] = '\0';
X }
X else
X {
X if ( cflag ) printf( "%d %d", i, k );
X close( fd );
X return( i );
X }
X }
X if ( wflag )
X {
X m = 0;
X for ( n=0; n < 8; n++ ) name[n]=entry.text[m++];
X name[n]='\0';
X for ( n=0; n < 13; n++ ) id[n]=entry.text[m++];
X id[n]='\0';
X for ( n=0; n < 24; n++ ) comment[n]=entry.text[m++];
X comment[n]='\0';
X printf(
X "%3d %-8s %-24s %4d %6d %6d\n", i,
X name, comment, entry.files, entry.start,
X entry.bytes );
X }
X if ( i == j )
X {
X
X sprintf( buf1,"%d:%d",
X entry.start, entry.bytes );
X for ( m = 0, p = buf1; *p; m++, p++ );
X *p++=m;
X *p = '\0';
X fd2 = open( "/tmp/ss.osets",
X O_WRONLY | O_CREAT, 0666 );
X write( fd2, buf1, strlen( buf1 ) );
X close( fd2 );
X if ( cflag )
X {
X printf( "%d %d", i, k );
X close( fd );
X return( i++ );
X }
X return( i );
X }
X k = entry.start + entry.bytes;
X }
X return( i );
X}
X
Xint r_info( num )
Xint num;
X
X{
X int i, k;
X k = num;
X for ( i = 1; i <= 9 ; i++ )
X {
X read( fd, &entry, sizeof(entry) );
X if ( entry.text[0] < '\0' ) break;
X if ( num == -1 ) k = i;
X if ( i == k ) printf( "%d %d %d", i,
X entry.start, entry.bytes );
X }
X return( i );
X}
X
Xint greatest()
X
X{
X int i, j, k;
X k = 0;
X for ( i = 1; i <= 9 ; i++ )
X {
X read( fd, &entry, sizeof(entry) );
X if ( entry.text[0] < '\0' ) break;
X j = atoi( entry.bytes );
X if ( j > k ) k = j;
X }
X return( k );
X}
END_OF_read.vtoc.c
if test 3698 -ne `wc -c <read.vtoc.c`; then
echo shar: \"read.vtoc.c\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f save.list -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"save.list\"
else
echo shar: Extracting \"save.list\" \(365 characters\)
sed "s/^X//" >save.list <<'END_OF_save.list'
X# This is the list of directories or files to be scanned
X# for 'squirrel'ing to 'arcdir'. Later, 'arcdir' is to be
X# processed by 'prep.sh' for transfer to tape.
X
X/news/to.zebra # This should be changed to your local name.
X/news/unix-pc
X/news/comp/sys/att
X/news/comp/sources
X
X# The following entries may be needed for converting old tapes.
X#/news/.tape.ops
X#/tmp
END_OF_save.list
if test 365 -ne `wc -c <save.list`; then
echo shar: \"save.list\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f squirrel.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"squirrel.sh\"
else
echo shar: Extracting \"squirrel.sh\" \(4537 characters\)
sed "s/^X//" >squirrel.sh <<'END_OF_squirrel.sh'
X# The purpose of this script is to 'squirrel' away files in the news
X# files for transfer to tape. By linking these files to entries in
X# the 'arcdir', they are protected from 'expire' yet do not use up an
X# great amount of disk space.
X
Xif [ ! "$WRKDIR" ]
Xthen
X . /news/.tape.ops/header.sh
Xfi
X
Xcd $WRKDIR
X
X# To commence, let's find out the status of our files and generate
X# the directories and file entries which are missing.
X
Xif [ ! -d $ARCDIR ]
Xthen
X mkdir $ARCDIR
X chmod 755 $ARCDIR
Xfi
X
Xif [ ! -s $PARAMS ]
Xthen
X > $PARAMS
X chgrp $GRP $PARAMS
X chown $OWN $PARAMS
X fnum = 1
Xelse
X fnum=`sed -n 's/Next fnum to link:\ //p' $PARAMS`
Xfi
X
X# Check that the 'params' file has the proper entries.
Xawk -F" " - '{
X if ( $0 ~ /Next fnum to link:/ ) ok1 = 1
X if ( $0 ~ /Number of bytes held:/ ) ok2 = 1
X if ( $0 ~ /Date last squirreled:/ ) ok3 = 1
X if ( $0 ~ /Name of work tape:/ ) ok4 = 1
X if ( $0 ~ /Blocks of offset:/ ) ok5 = 1
X if ( $0 ~ /Date last taped:/ ) ok6 = 1
X if ( $0 ~ /Blocks available:/ ) ok7 = 1
X if ( $0 ~ /Next chapter number:/ ) ok8 = 1
X if ( $0 ~ /Compression factor:/ ) ok9 = 1
X }
X END{
X if ( ok1 != 1 ) print "Next fnum to link: 0"
X if ( ok2 != 1 ) print "Number of bytes held: 0"
X if ( ok3 != 1 ) print "Date last squirreled: 0"
X if ( ok4 != 1 ) print "Name of work tape: News-000"
X if ( ok5 != 1 ) print "Blocks of offset: 0"
X if ( ok6 != 1 ) print "Date last taped: 0"
X if ( ok7 != 1 ) print "Blocks available: 45566"
X if ( ok8 != 1 ) print "Next chapter number: 0"
X if ( ok9 != 1 ) print "Compression factor: 1000"
X }' $PARAMS >> $PARAMS
X
Xtoday=`date '+%y-%m-%d'`
X
X# If we have a 'link.list' file, remove entries over two months old
Xif [ -s $LINKS ]
Xthen
X cp $LINKS $KLINKS
X > $TMP
X awk - 'BEGIN{ now = "'"$today"'"
X split( now, old, "-" )
X old[2] -= 2
X if ( old[2] < 1 ){
X old[1] -= 1
X old[2] += 12
X }
X if ( old[1] < 70 ) old[1] += 100
X daysofmonth = "31 28 31 30 31 30 31 31 30 31 30 31"
X split( daysofmonth, days )
X if ( old[3] > days[ old[2] ] ) old[3] = days[ old[2] ]
X FS = " "
X }
X { split( $2, was, "-" )
X if ( was[1] < 70 ) was[1] += 100
X if ( old[1] > was[1] ){ next }
X if ( old[1] == was[1] && old[2] > was[2] ){ next }
X if ( old[1] == was[1] && old[2] == was[2] && \
X old[3] > was[3] ){ next }
X print $0 >> "'$TMP'"
X }' $KLINKS
X mv $TMP $KLINKS
Xelse
X > $KLINKS
X > $LINKS
Xfi
X
Xx=fnum
Xwhile [ -f $ARCDIR/$x ]
Xdo
X rm -f $ARCDIR/$x
X (( x = x + 1 ))
Xdone
X
Xrm -f $WRKDIR/contents
Xrm -f $GOOFS
X
Xfunction recur
X{
X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ]
X then
X return -1
X fi
X typeset this next
X this=$1
X set `ls -1 $1`
X while [ $1 ]
X do
X next=$this/$1
X if [ -d $next ]
X then
X recur $next
X fi
X if [ -f $next ]
X then
X id=`sed -n '/^Message-ID:/ {
X s/Message-ID: //p
X q
X }' $next | tr -d "/*?^$\[\]|"`
X if [ -n "$id" ]
X then
X if
X [ -z "`sed -n '/'"$id"'/p' $KLINKS`" ]
X then
X if ln $next $ARCDIR/$fnum
X then
X let fnum="fnum+1"
X echo "$id""\t"$today >> $KLINKS
X else
X echo $next", "$ARCDIR/$fnum >> $GOOFS
X fi
X fi
X fi
X fi
X shift
X done
X}
X
Xwhile
X read fname
Xdo
X dname=`echo $fname | cut -f1 -d"#"`
X if [ -n "$dname" ]
X then
X recur $dname
X fi
Xdone <$LIST
X
Xfunction commas
X{ echo `echo $1 | awk - '{ n = length( $0 )
X s = n % 3
X if ( s == 0 ) s = 3
X m = substr( $0, 1, s )
X s++
X while ( n > s ){
X o = substr( $0, s, 3 )
X m = m","o
X s += 3
X }
X print m }' -`
X}
X
Xnbytes=`ls -l $ARCDIR | awk - '{ sum = sum + $5 }
X END{ print sum }' - `
Xcbytes=`commas $nbytes`
X
Xsort -d -o $LINKS $KLINKS
X
X> $TMP
Xawk -F" " - '{ if ( $0 ~ /Next fnum to link:/ ) $2 = "'"$fnum"'"
X if ( $0 ~ /Number of bytes held:/ ) $2 = "'"$cbytes"'"
X if ( $0 ~ /Date last squirreled:/ ) $2 = "'"$today"'"
X print $1" "$2 >> "'$TMP'"
X }' $PARAMS
Xmv $TMP $PARAMS
Xchgrp $GRP $PARAMS
Xchown $OWN $PARAMS
Xchgrp $GRP $LINKS
Xchown $OWN $LINKS
Xrm -f $KLINKS
X
Xoffset=`sed -n 's/Blocks of offset:\ //p' $PARAMS`
Xchap=`sed -n 's/Next chapter number:\ //p' $PARAMS`
Xfactor=`sed -n 's/Compression factor:\ //p' $PARAMS`
X
X(( min_size = ( MAXOFFSET - offset ) / (10 - chap) ))
X(( this = nbytes/factor )) # blocks for this collection
X
Xecho "To: netnews
XSubject: File Squirreling.\n
XYour 'savdir' currently has $cbytes bytes of news articles saved.
X Recommended size of next tape entry: $min_size blocks
X Approximate size of this collection: $this blocks" > $TMP
Xif [ "$this" -gt "$min_size" ]
Xthen
X echo "\t\tYou better transfer these to tape soon.!" >> $TMP
Xfi
Xecho "\nThis is from your friendly file saver." >> $TMP
Xcat $TMP | /bin/rmail netnews
Xrm $TMP
END_OF_squirrel.sh
if test 4537 -ne `wc -c <squirrel.sh`; then
echo shar: \"squirrel.sh\" unpacked with wrong size!
fi
chmod +x squirrel.sh
# end of overwriting check
fi
if test -f strt.hdr.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"strt.hdr.sh\"
else
echo shar: Extracting \"strt.hdr.sh\" \(1164 characters\)
sed "s/^X//" >strt.hdr.sh <<'END_OF_strt.hdr.sh'
X
X# The purpose of this script is to prepare the 'squirrel'ed files
X# for transfer to tape. This consists of building a 'contents'
X# file from the "Subject" line in the posted articles. The 'contents'
X# is sorted and similar titles are moved together for easier reference.
X# The articles are then compressed and moved to the 'savdir' where
X# they can copied to tape.
X
Xexport NEWSDEV=/dev/fp002
Xexport WRKDIR=/news/.start
Xexport CACHE=$WRKDIR/cache
Xexport ARCDIR=$WRKDIR/arcdir
Xexport RDPARMS=$WRKDIR/rd.params
Xexport PARAMS=$WRKDIR/wr.params
Xexport SAVDIR=$WRKDIR/savdir
Xexport CONFILE=$WRKDIR/contents
Xexport SRCDIR=/news/.tape.ops/src
Xexport PNTRS=$WRKDIR/pntrs
Xexport CPNTRS=$WRKDIR/c.pntrs
Xexport LIST=$WRKDIR/save.list # Directory list to be saved
Xexport LINKS=$WRKDIR/link.list
Xexport KLINKS=$WRKDIR/wrk.links
Xexport GOOFS=$WRKDIR/goofs
Xexport TMP1=$WRKDIR/tmpp1
Xexport TMP2=$WRKDIR/tmpp2
Xexport COPY=0
Xexport GRP=news
Xexport OWN=netnews
Xexport PERCENT=6200 # This is the last five percent of disk space
Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes
X# If we add 2 blocks for header, B200 512 byte blocks
X# or 1DAA blocks per stream???
END_OF_strt.hdr.sh
if test 1164 -ne `wc -c <strt.hdr.sh`; then
echo shar: \"strt.hdr.sh\" unpacked with wrong size!
fi
# end of overwriting check
fi
echo shar: End of archive 1 \(of 2\).
cp /dev/null ark1isdone
MISSING=""
for I in 1 2 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have unpacked both archives.
rm -f ark[1-9]isdone
else
echo You still need to unpack the following archives:
echo " " ${MISSING}
fi
## End of shell archive.
exit 0
--
Vernon C. Hoxie {ncar,nbires,boulder,isis}!scicom!zebra!vern
3975 W. 29th Ave. voice: 303-477-1780
Denver, Colo., 80212 TB+ uucp: 303-455-2670
More information about the Unix-pc.sources
mailing list