Xenix 286 and 386 crypt bugs.

[Keith Gabryelski]

A message similar to this one was posted to comp.bugs.sys5 by Michael
Ditto (Ford at kenobi.UUCP).  This message, however, is directed to SCO
Xenix 2.2.1 users (286 and 386).  At the end of this message, there is
information on how to obtain the binary patches for SCO Xenix.

There is a bug in the crypt(S) library function in all Xenix libraries
(and most System V libraries) which completely prevents proper
encryption/decryption of data using the DES algorithm.  The routines
affected are setkey() and encrypt(), the crypt() function works as it
is.

Enclosed is a program, "crypttest", which you can use to see if your
DES routines work.  The program encrypts and decrypts a string
("testdata") and prints the result in binary.  The correct result is:

00101110 10100110 11001110 00101110 00100110 10000110 00101110 10000110 

The incorrect result is:

11011000 00111011 11100100 10011100 00000010 01111000 10010101 10000111

The problem in the crypt library is that a small chunk of code is in
the wrong place.  I don't have access to Xenix source code, so I can
only give a basic description based on analysis of the object code.

At the very end of the function 'setkey' there should be a loop that
copies the "e2" array into the "E" array (These symbols gotten from
SysV binaries and may be different on Xenix).  For some reason, the
crypt.o in the library has this code moved from the end of setkey()
into the middle of the function crypt().  It is shortly after the call
to setkey(), so crypt() still works correctly, but direct calls to
setkey() will never have the necessary code executed.

This is really strange, a whole chunk of code was moved into an
entirely different function.  Anybody have an explanation?

The code in question looks something like this:

	for ( i=0 ; i<48 ; ++i )
	    E[i] = e2[i];

If you have source code, find the loop above just after the call to
setkey() in the crypt() function and MOVE it to the very end of the
setkey() function.

The bug was found by myself and Michael Ditto (ford at kenobi) by
comparing a working version (bsd) against the binaries of several SysV
and Xenix systems.

We have seen the bug in the libraries on the AT&T Unix PC and in SCO
Xenix, so I would assume it is in all SysV-derived libraries.

Mike has posted a binary patch for the Unix PC to the appropriate
newsgroups and this post should be followed by two shar file posts.
One is the binary patch for 386 systems and the other for 286 systems.
These patches fix the libraries for all models on their respected
systems.

If the next two shar files just don't make it across the net, these three
shar files (for the 286, 386, and Unix PC) will be available for anonymous
uucp from portnoy.uucp.

For these patches to work, you must have the crypt libraries supplied
by SCO.  They are not given with the standard operating system package
and must be asked for explicitly.  (They are sent free of charge to
anyone with a SCO license in the USA)

To download:
	
portnoy Any ACU 2400 16194580259 ogin:--ogin: uucp
portnoy Any ACU 1200 16194580259 "" \r\d\r\c ogin: uucp

The files:
	~/crypt-286.shar	(SCO Xenix 286 hack)
	~/crypt-386.shar	(SCO Xenix 386 hack)
	~/crypt-upc.shar	(Unix PC hack -- thank you ford)

AT&T and SCO will be sent bug reports.

Here's crypttest.c:


#define PASSWORD "password"
#define DATA "testdata"


main()
{
    char key[64], block[64];
    char buf[9];

    makeblock(block, DATA);
    puts("Here is the data");
    dumpblock(block);

    makeblock(key, PASSWORD);
    puts("Here is the key");
    dumpblock(key);

    setkey(key);

    encrypt(block, 0);
    puts("Here is the encrypted data");
    dumpblock(block);

    encrypt(block, 1);
    puts("Here is the decrypted data");
    dumpblock(block);

    unmakeblock(buf, block);
    buf[8] = '\0';

    printf("The resulting string is: `%s'\n", buf);

    return strcmp(buf, DATA);
}


dumpblock(block)
char *block;
{
    register i, j;
    for ( i=0 ; i<8 ; ++i )
    {
	for ( j=0 ; j<8 ; ++j )
	    putchar('0' + *block++);
	putchar(' ');
    }
    putchar('\n');
}


/* converts a 8-byte string into a 64-byte */
/* array of 0-or-1 bytes suitable for encrypt(3C) */
makeblock(array, string)
char *array, *string;
{
    register unsigned char c;
    register int i, j;

    for ( i=0 ; i<8 ; ++i )
    {
	c = *string++;
	for ( j=0 ; j<8 ; ++j )
	{
	    *array++ = c&1;
	    c >>= 1;
	}
    }
}


/* converts a 64-byte array of 0-or-1 bytes as */
/* returned by crypt(3C) into a 8-byte string */
unmakeblock(string, array)
char *string, *array;
{
    register int j, i;
    register unsigned char c;

    for ( i=0 ; i<8 ; ++i )
    {
	c = 0;
	for ( j=0 ; j<8 ; ++j )
	{
	    c >>= 1;
	    if (*array++)
		c |= 0x80;
	}
	*string++ = c;
    }
}
-- 
[  Keith   ]  UUCP: {cbosgd, hplabs!hp-sdd, sdcsvax, nosc}!crash!portnoy!ag
[Gabryelski]  INET: ag at portnoy.cts.com      ARPA: crash!portnoy!ag at nosc.mil