Subject: bug in and fix to crypt(3C)

[Mike Ditto]

There is a bug in the crypt(3C) library function in most (if not all)
System V libraries which completely prevents proper encryption/
decryption of data using the DES algorithm.  The routines affected are
setkey() and encrypt(), the crypt() function works as it is.


Enclosed is a program, "crypttest", which you can use to see if your
DES routines work.  The program encrypts and decrypts a string
("testdata") and prints the result in binary.  The correct result is:

00101110 10100110 11001110 00101110 00100110 10000110 00101110 10000110 

The incorrect result is:

11011000 00111011 11100100 10011100 00000010 01111000 10010101 10000111


The problem in the crypt library is that a small chunk of code is in
the wrong place.  I don't have access to SysV source, so I can only
give a basic description based on analysis of the object code.

At the very end of the function 'setkey' there should be a loop that
copies the "e2" array into the "E" array.  For some reason, the
crypt.o in the library has this code moved from the end of setkey()
into the middle of the function crypt().  It is shortly after the call
to setkey(), so crypt() still works correctly, but direct calls to
setkey() will never have the necessary code executed.

The code in question looks something like this:

	for ( i=0 ; i<48 ; ++i )
	    E[i] = e2[i];

If you have source code, find the loop above just after the call to
setkey() in the crypt() function and MOVE it to the very end of the
setkey() function.  Perhaps someone with the AT&T source can make the
change and post a context diff, so other people can use the patch
program to fix it.

The bug was found by myself and Keith Gabryelski (ag at portnoy.CTS.COM)
by comparing a working version against the binaries of several SysV
systems, using only our bare hands and adb (Well, with some help from
M-x compare-windows, etc.).

We have seen the bug in the libraries on the AT&T Unix PC and in SCO
Xenix, so I would assume it is in all SysV-derived libraries.  I have
posted a binary patch for the Unix PC; Keith will post a binary patch
for SCO Xenix later (it's more difficult, what with the 286/s/m/l &
386 libraries).  We don't beleive in waiting for the turnaround
involved in O.S. updates :-) :-) but AT&T and SCO will be sent bug
reports.

					-=] Ford [=-

"Once there were parking lots,		(In Real Life:  Mike Ditto)
now it's a peaceful oasis.		ford%kenobi at crash.CTS.COM
This was a Pizza Hut,			...!sdcsvax!crash!kenobi!ford
now it's all covered with daisies." -- Talking Heads


Here's crypttest.c:


#define PASSWORD "password"
#define DATA "testdata"


main()
{
    char key[64], block[64];
    char buf[9];

    makeblock(block, DATA);
    puts("Here is the data");
    dumpblock(block);

    makeblock(key, PASSWORD);
    puts("Here is the key");
    dumpblock(key);

    setkey(key);

    encrypt(block, 0);
    puts("Here is the encrypted data");
    dumpblock(block);

    encrypt(block, 1);
    puts("Here is the decrypted data");
    dumpblock(block);

    unmakeblock(buf, block);
    buf[8] = '\0';

    printf("The resulting string is: `%s'\n", buf);

    return strcmp(buf, DATA);
}


dumpblock(block)
char *block;
{
    register i, j;
    for ( i=0 ; i<8 ; ++i )
    {
	for ( j=0 ; j<8 ; ++j )
	    putchar('0' + *block++);
	putchar(' ');
    }
    putchar('\n');
}


/* converts a 8-byte string into a 64-byte */
/* array of 0-or-1 bytes suitable for encrypt(3C) */
makeblock(array, string)
char *array, *string;
{
    register unsigned char c;
    register int i, j;

    for ( i=0 ; i<8 ; ++i )
    {
	c = *string++;
	for ( j=0 ; j<8 ; ++j )
	{
	    *array++ = c&1;
	    c >>= 1;
	}
    }
}


/* converts a 64-byte array of 0-or-1 bytes as */
/* returned by crypt(3C) into a 8-byte string */
unmakeblock(string, array)
char *string, *array;
{
    register int j, i;
    register unsigned char c;

    for ( i=0 ; i<8 ; ++i )
    {
	c = 0;
	for ( j=0 ; j<8 ; ++j )
	{
	    c >>= 1;
	    if (*array++)
		c |= 0x80;
	}
	*string++ = c;
    }
}