benchmarking packages

[Ed Arnold]

Several months ago, I posted an inquiry for information on benchmarking
packages; the information received has been condensed into this posting.
In some cases, it wasn't possible to follow a lead as far as an address
and phone number.  If anyone out there has any further leads on any of the
following material, please notify me; I will respost this item if further
information comes to light.

CAVEAT: there may be inaccuracies, or plain expressions of opinion,
in this information.  In particular, if you are a vendor and are
insulted by something you read here, there is no point in calling
me to chew me out!  If you care to e-mail me more correct or complete
information on your product, then I will repost a correction of this
article.

ACQUISITION SUPPORT

I've come across one company which claims to offer a "total acquisition
support" service, i.e. RFP writing and benchmarking independent of any
vendor.  I suspect there are others who do this sort of thing, however,
I have not yet run across them.

o A&T Systems, Silver Spring MD; Marcus Rogerson, 301-384-1425
o Vendors: almost all vendors of any size will have someone who is
  a specialist in performance analysis.  They will have run one or
  more "standard" benchmarking suites on their equipment, and will be
  happy to provide you with this information.  Whether you choose to
  believe it is, of course, up to you; if you take this approach, be
  sure you understand how settable parameters were likely to have
  affected the result, and whether it is possible that some other
  settings might have resulted in a complete turnaround of the results
  with regard to comparing two systems.  (Easier said than done. :-))

VALIDATION SUITES

o SVVS: the System V Validation Suite is available directly from AT&T
  technology licensing in North Carolina, 800-828-unix.  However, you
  must be a S/V source licensee to obtain/use this.
o POSIX: there is a company working with the National Bureau of Standards
  to construct a validation suite for POSIX.  I haven't yet found out
  who that is.
o Perennial: Perennial Inc. (4677 Old Ironsides Drive #450, Santa Clara, CA,
  95054; 408-727-2255) has available validation suites for System V.x
  and BSD 4.x, for C compilers, and some performance benchmarking stuff.
o BRL: Doug Gwyn of BRL (gwyn at brl.arpa) has a small suite of System V.2
  verification and load testing programs, built for an Army procurement,
  which is available for free.
o Certification thru AT&T or CSRG: vendors who claim to be compliant with
  System V.x should have a certificate on file with the AT&T compliance
  testing group.  In addition, one reference on the net claims that the
  Berkeley CSRG has a similar program for BSD4.x, although I'm not
  personally aware this is so.  Anybody know?

BENCHMARKING SUITES

The following benchmark suites are available.  In some cases,
I haven't yet determined how to obtain the item.

o Monash University (MUSBUS) benchmark suite, which has been posted to
  Usenet in comp.sources.unix.  This suite was authored by Ken J. McDonnell,
  now with Pyramid Technology: kenj at pyrnova.pyramid.com.  It may be
  obtained from uunet.uu.net via anonymous ftp, or via the archive server
  at uunet!netlib.  The 5.0 version is in comp.sources.unix/volume11/musbus;
  the patch for version 5.2 is in comp.sources.unix/volume12/musbus5.2.
    In personal correspondence with the author, he stated that MUSBUS does
  a better job than some of the older suites, particularly with regard to
  imposing a workload of real programs and finding a system's point of
  resource depletion.
o Purdue benchmark suite, from Purdue University.  Although a correspondent
  claimed there is such a thing, I have not been able to obtain a copy
  of it, or a firm reference.
o BRLCAD, available free of charge from Ballistics Research Lab.
  Info on this may be obtained on request from cad at brl.arpa.
o ditroff: one correspondent claims that multiple copies of ditroff running
  are, in his experience, as good a way of any at testing cpu and i/o
  performance.
o ENEA suite: probably available for free from Mats Josefsson of Enea Data,
  Taeby, Sweden.  Mail uunet!enea!mats to find out about this.
o Linpack, Dhrystone, Whetstone: these traditional benchmarking tools
  are available from the archive server, netlib at anl-mcs.arpa (research!netlib).
  It is my understanding that these tools aren't very good at measuring
  the performance of large, multi-processor systems; they are best for
  measuring traditional single-CPU architectures, or for those who are
  more concerned with flat-out floating point performance.
    To find out more about what's available from netlib, mail to the
  address given, with a message body that contains "send index".
o AIM: AIM Technology, 3350 West Bayshore Road #203, Palo Alto CA 94303;
  415-856-8649 makes available a package called the AIM Benchmarks,
  which has been around for a number of years.  I believe the package
  is now referred to as "AIM 3".
o We at NCAR have a home-brew suite that we built in 1985; this was to
  support an acquisition effort that ended in the purchase of two
  Pyramid 90x machines.  If you want a copy of this, mail me
  (era at ncar.ucar.edu, ncar!era).
o University of Texas: Michael Cerda at U. Texas (cerda at emx.utexas.edu)
  supposedly has a good suite of benchmarks called "sequent.sh".
  I have not yet seen this, however.
o Workstation Laboratories: it is my understanding that this company offers
  a product similar to AIM, tailored to the workstation market.  However,
  I have not yet determined an address or telephone number.
o The Usenet management/archive node, uunet.uu.net, may have items for
  benchmarking, other than MUSBUS mentioned above.  Interested users
  may use anonymous ftp, or mail to uunet!netlib ("send index"), to
  investigate this.
o There is a benchmark named "TP-1", designed to simulate on-line
  transaction processing in banking, database access, and similar
  environments.  This benchmark is typically used on big systems,
  such as Tandem networks, 3090s, and the like.  Apparently, TP-1
  is rather sensitive to the parameters with which it is setup.
  I have not yet determined if/where this suite may be obtained.

SYSTEM PERFORMANCE EVALUATIONS

There are several companies/organizations which provide information
from running their particular benchmark suite, on "typical" configurations
hardware from many different vendors.

o BRLCAD includes stats for many machines, from small Suns to Cray.
o AIM: (see address above) AIM makes available "Unix Performance Reports",
  the results of the AIM Benchmarks run on a large number of different
  machines, such as Sun, Pyramid, and Sequent.  The data provided is
  broken down into categories of subsystem and application performance.
  Three of their reports can currently be obtained for free; each
  additional machine report is $25.
o Neal Nelson Associates: Chicago, 312-332-3242.  Apparently, their suite
  is sometimes referred to as the "Neal Nelson Business Benchmark."
  They offer source leasing of their suite, canned performance reports
  (one machine against another, for $96), and custom testing.  In a
  conversation with Karen McBride of this company, she claimed that they
  have worked with a number of government agencies in developing averaged
  numbers (from their performance reports) for inclusion in RFPs, and will
  provide references into those agencies on request.
    I have read that the comments above about Linpack & friends are also
  applicable to the Nelson suite, i.e., it is better suited to measuring
  traditional single-CPU architectures, than large multi-processor systems.
  However, I have no personal knowledge of whether or not this is so.
--------
Ed Arnold * NCAR (Nat'l Center for Atmospheric Research) * Mesa Lab
PO Box 3000 * Boulder, CO  80307-3000 * 303-497-1253
era at ncar.ucar.edu [128.117.64.4] * {ames,gatech,noao,...}!ncar!era