Report on 1003.3: POSIX Test Methods and Conformance

[Peter Collinson]

Submitted-by: pc at hillside.co.uk (Peter Collinson)

USENIX Standards Watchdog Committee
Stephen R. Walli <stephe at usenix.org>, Report Editor
Report on 1003.3: POSIX Test Methods and Conformance


Andrew Twigger <att at root.co.uk> reports on the April 15-19,
1991 meeting in Chicago, IL:

Summary

The POSIX.2 (Shell and Utilities) working group made good progress
writing test assertions this week, with POSIX.3's (Test Methods and
Conformance) help. Many working groups, however, are discovering that
writing test assertions requires a non-trivial effort. This week also
saw the delivery of the newly published ``IEEE 1003.3-1991 - Test
Methods for Measuring Conformance to POSIX''. Concerns are still being
raised over NIST's certification policies.

Report

Chicago will probably go down in history as the meeting where test
methods invaded the POSIX working groups with a vengeance.  After
years of mild abuse and jesting (mostly aimed at NIST), the SCCT
(Steering Committee on Conformance Testing) seems to be succeeding in
the goal of ensuring that POSIX standards are balloted with test method
specifications.  Despite rumours during the week that a wake had been
arranged for the SCCT Chair, most of the screams were heard from
working groups, who having been previously informed that test methods
would be easy to write and would only take a couple of meetings, were
finding that this was a far from straightforward task.

While most of the remaining members of the original POSIX.3 working
group continued work with the remaining members of POSIX.2 in
generating assertions for the POSIX.2 standard, a few of the POSIX.3
elders started helping other working groups to develop test methods
for their standards.  The POSIX.3.2 group (i.e POSIX.3 + POSIX.2) met
for three days during the week and spent all of that time writing
assertions in small groups of three or four people.

Some of the more difficult aspects of POSIX.2 were tackled,
specifically Basic Regular Expressions and the Make utility.  Most of
the smaller utilities have assertions written already although most of
these need to be updated to align with the current draft.  It is hoped
that enough of the work will have been completed after the October
balloting commencing in the first half of 1992.

Other working groups that have started producing test methods include
POSIX.4, POSIX.6, POSIX.8, POSIX.15, POSIX.17 and P1224.1, P1224.2.
Most of these groups are at an early stage in their test method
development and are producing a wide variety of problems for the
``experts'' to address.  Several of these groups have noted that the
formal process of producing test assertions has uncovered a variety of
deficiencies in their drafts; so perhaps there is some benefit in test
methods after all!

The highlight of the week was the arrival of the latest of the series
of POSIX standards, IEEE 1003.3-1991.  This document was made
available at the extraordinary discounted price of $15.00 per copy,
which works out to 30 cents a page!  Still I suppose that considering
the number of committee hours that went into the document, it's a real
bargain.  (One working group member calculated an industry cost in
excess of $5,000 per page.)

Other concerns which arose during the week relate to NIST's adopted
certification policies and procedures.  Many working groups continue
to be concerned about these. This has been a long running battle
involving both accredited testing centres and implementation suppliers
in assisting NIST in the refining of their policies.

The current major cause for concern is whether there would be equality
in the certification process or whether a particular implementor would
gain advantage from receiving the first conformance certificate.  NIST
was not explicit as to the procedures that they would employ to deal
with the initial surge of certification requests, but made assurances
that everybody would be satisfied when the process was completed.
This seemed to satisfy nobody!  We'll have to wait until Santa Clara
to see whether NIST is really here to help us.


Volume-Number: Volume 24, Number 23