Standards Update, IEEE 1003.2: Shell and Utilities

[Peter Collinson]

Submitted-by: pc at hillside.co.uk (Peter Collinson)

USENIX Standards Watchdog Committee
Stephen Walli <stephe at usenix.org>, Report Editor
Report on IEEE 1003.2: Shell and Utilities


David Rowley <david at mks.com> reports on the April 15-19 meeting in
Chicago, IL:

Background

A brief POSIX.2 project description:

   - POSIX.2 is the base standard and deals with the basic shell
     programming language and a set of utilities required for the
     portability of shell scripts.  It excludes most features that
     might be considered interactive.  POSIX.2 also standardizes
     command-line and function interfaces related to certain POSIX.2
     utilities (e.g., popen(), regular expressions, etc.).  This part
     of POSIX.2, which was developed first, is sometimes known as
     ``Dot 2 Classic.''

   - POSIX.2a, the User Portability Extension or UPE, is a supplement
     to the base POSIX.2 standard and standardizes commands, such as
     vi, that might not appear in shell scripts but are important
     enough that users must learn them on any real system.  It is
     essentially an interactive standard, and it will eventually be an
     optional chapter to a future draft of the base document.  This
     approach allows the adoption of the UPE to trail Dot 2 Classic
     without delaying it.

     Some utilities have both interactive and non- interactive
     features.  In such cases, the UPE defines extensions from the
     base POSIX.2 utility.  Features used both interactively and in
     scripts tend to be defined in the base standard.

   - POSIX.2b is a newly approved project which will cover extensions
     and new requests from other groups, such as utilities for the
     POSIX.4 (Realtime) and POSIX.6 (Security) documents.

Together, Dot 2 Classic and the UPE will make up the International
Standards Organization's ISO 9945-2 -- the second volume of the
proposed ISO three-volume POSIX standard.

Summary

POSIX.2 (Shell and Utilities) closed its recirculation ballot on 29
March. The Chair feels there will only be a small number of changes to
the current draft, probably circulated as change pages. There were
some ballot concerns over the internationalization areas, but these
will likely remain intact due to current support. There was also a
concern raised over the ballot period for a 900+ page document.
POSIX.2a closed its recirculation ballot on 24 April.

POSIX.2b has been approved after a number of scope change
recommendations from the PMC.  The POSIX.2 group requests technology
for both a new archive format, and also a new family of compression
utilities.  Much of the Chicago meeting was spent with POSIX.3.2 (Test
Methods for POSIX.2) creating test assertions for the document.

Status of POSIX.2 Balloting

The Draft 11 Recirculation Ballot for POSIX.2 closed March 29th. Some
balloters seemed to forget that it was a recirculation ballot, as
there were a large number of objections on items other than changes.
These were ruled unresponsive.

Hal Jespersen, the POSIX.2 Chair and Technical Editor, believes that
there will only be a small number of actual modifications to the
draft. Draft 12 (which may possibly be called Draft 11.1) will likely
be distributed as a set of changed pages, which he estimates to number
about 200.  (More recent estimates suggest the number of pages to be as
low as 50). Expect it sometime around July.

There were a number of objections to the internationalization part of
POSIX.2, but since Hal has support from X/Open, WG15, etc, he thinks
the current specification will remain largely intact.

There was a problem with the Draft 11 distribution. POSIX.2 is now
over 900 pages. It's balloting period was 30 days, although with a
mailing lead it was about 6 weeks.  Due to postal services, some
members of the balloting group only received their ballot copies two
weeks prior to the closing deadline. Hal Jespersen was as
accommodating as he could be on the deadline, but the extent of these
submissions was definitely affected.  The question rears its head
again.  Should we be balloting POSIX standards the same as we ballot
smaller hardware standards?

The ISO standards process sees a document move through three phases on
its way to standardization -- Committee Document, Draft International
Standard, and finally International Standard.  Draft 9 of POSIX.2 is
currently being used as a committee document.  The ISO has requested
the U.S. Member Body to forward to them another draft once it has
become more stable. The next draft (Draft 12 or Draft 11.1) will be a
likely candidate.  The ISO has delegated responsibility for producing
the POSIX draft standards documents to the U.S. Member Body, ANSI,
which in turn delegated the responsibility to the IEEE.

Status of POSIX.2a Balloting

The Draft 6 Recirculation Ballot of POSIX.2a (UPE) closed April 24th.
Unfortunately the deadline for comments came a mere three days after
the end of the April meeting. There were quite a few comments on the
unfortunate timing of the ballot close. Work on ballot resolution is
ongoing.

New PARs

The Project Management Committee (PMC) has recommended the proposed
PAR (Project Authorization Request) for 1003.2b be split into two
parts. One part will cover extensions and new requests from other
groups, such as the tar, cpio and new pax file formats from POSIX.1
(Kernel) and utilities from POSIX.4 (Realtime) and POSIX.6
(Security).  The timing and alignment issues with the ISO 9945-2
balloting process will be covered by the other part.

The scope of this second PAR is restricted to standardization of
existing industry practice. The group does not want to get into
designing new utilities.

There is also concern over draft stability when discussing the
commands to access the features from the POSIX.4 and POSIX.6
standards. How mature does the feature have to be before the POSIX.2
group goes to the effort of defining a command interface to it ?

Discussion

Donn Terry, the chair of POSIX.1 officially handed off responsibility
of the pax file formats, including the new format currently being
designed, to the POSIX.2 group.

A proposal was examined to reserve the utility status return code 126
to indicate that a utility was found but could not be successfully
invoked.  This would be especially useful in systems with limited
resources, where execution can not be assured even though the utility
has been found. The group generally agreed that this was reasonable.

There was a question as to whether the warning message for getopts
should be specified in the standard or not, so that filters could
parse it. It was decided to not specify the error format, since there
is no precedent for stating the format of something written to
standard error.

There was discussion on removing -e from pax, since charmaps were not
really intended to be used in this manner.  The -e option was intended
to allow filenames to be written out using only characters from the
portable character set.  OSF had already implemented this in their
pax, and agreed that it didn't work out too well.

The $(( notation in the Korn Shell currently can have two widely
different meanings: either spawning a subshell or expressing an
arithmetic operation. The working group agreed that disambiguating by
placing a space between the two parentheses, though inelegant, was the
best approach.

There was some discussion on a proposal on User Controllable Limits,
and whether or not it had relevance to POSIX.2.  The group felt that
there should be a command interface to these services, with the
functional interface to be provided by POSIX.1.  A proposal for the
POSIX.2 interface is now being solicited.

We also discussed the the test command. David Korn proposed fixing the
functionality of test based on the number of arguments given (1, 2 or
3). Invocations with greater than 3 arguments would not be portable.
We generally agreed on this approach.

Richard Hart from POSIX.7 (System Administration) presented the syntax
for a new printing command based on the MIT/Athena Palladium network
printing services. Everyone in the POSIX.2 group agreed that the
proposed syntax was awkward:

        prpr -print-quality draft

            means use draft if you can

        prpr =print-quality draft

            means you must use draft

        prpr =p-q draft

            means the same thing, but ``print-quality''
            has been abbreviated.

The abbreviation mechanism is particularly poor, since it is likely
that new extensions will cause namespace conflicts.

Requests for technology

There is an opportunity now to propose a new archive format.  The only
prerequisites are that it is ISO 1001 tape format compatible, and uses
the ISO 646 character set.  This consists of the invariant codeset
from a variety of character set standards, largely 7-Bit ASCII minus
about 10 contentious characters.  Here's a list of requirements:

   o Should be textual (mailable) if members are all textual

   o Should support filename and file contents mapping (eg.
     for dynamic encryption or compression)

   o Should be extensible

Personally I don't understand why the ISO 1001 tape format needs to be
a restriction. Archive formats have many other uses besides tape
backup and transfer. To embed the tape format in what could otherwise
be a general-use archive format seems overly complex and restrictive.

The group is also looking for a new family of compression utilities,
now that the Lempel-Ziv-Welch family of commands have been removed
from the standard.  The main requirements for a substitute are:

   o The algorithm should be expressed (expressible) in a
     language independent form

   o The algorithm should be free of patent issues

Test Plans and Assertions

A test plan for POSIX.2 and POSIX.2a has been written, and has been
passed to POSIX.3.2 (Test Assertions for POSIX.2) for comment and
approval.

Tuesday to Thursday were spent writing test assertions in a joint
meeting between POSIX.2 and POSIX.3.2 Commands tackled included make,
regular expressions, ln, cp, rm, mv, pax, pathconf, echo and read.

Some members volunteered test assertions written by their companies,
usually to a previous draft. They were almost always unusable, either
because they were out of date (based on previous drafts), or of poor
quality. Writing good test assertions is very difficult, and quickly
points out (the many) ambiguous wordings in the draft.

The POSIX.3.2 group would like to go to a mock ballot after the
October meeting in Parsippany, New Jersey.







































Volume-Number: Volume 23, Number 97