Bug in 'setjmp/longjmp' for pdp11 (v7 and maybe more)

[Chris Maltby]

[]
There is a bug in setjmp/longjmp for the pdp11 (v7 at least),
which may be present for lots of similar architectures. The symptom
is a stack which grows every call, until a segmentation trap is
generated.
Try this program...

#include <setjmp.h>
jmp_buf	goop;

main()
{
	setjmp(&goop)
	fred(0, 0, 0, 0, 0);
#ifndef	REALLYBAD
	longjmp(&goop, 1);
#endif
}
fred(a)
{
	printf("%o\n", &a);
#ifdef REALLYBAD
	longjmp(&goop, 1);
#endif
}

The value printed out will decrease (increase) by the difference in
size between the arguments to the setjmp call, and the arguments to
the function called which stacked the environment that setjmp was called
in. In the above example, if REALLYBAD is undefined, the stack grows
by 2 every call, but if REALLYBAD is defined the growth is 6 per call.

The problem is due to the way longjmp unwinds the stack when trying
to find the frame that called setjmp. It simulates the return from
setjmp, but leaves the arguments to longjmp (or fred) still stacked.
The code right after the setjmp call only throws away 1 argument...

There are two fixes...
1)
	Always call setjmp with as many args as the function which
	will be unwound. EG for the above

	#ifdef	REALLYBAD
		setjmp(&goop, 0, 0, 0, 0);
	#else
		setjmp(&goop, 0);
	#endif

	I'd only recommend this if you don't have the source.

2)
	libc/gen/setjmp.s line 29 onwards
	Remove the lines surrounded by ----
	Add the lines surrounded by ++++

	1:
		cmp	(r5),(r1)
-------------------------------------------------
--		beq	1f
-------------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++++++
++		beq	2f
+++++++++++++++++++++++++++++++++++++++++++++++++
		mov	(r5),r5
		bne	1b
	/ panic -- r2-r4 lost
+++++++++++++++++++++++++++++++++++++++++++++++++
++		br	1f
++	2:
++		/ can't use cret as number or args may be different.
++		mov	r5,r2
++		mov	-(r2),r4
++		mov	-(r2),r3
++		mov	-(r2),r2
++	1:
+++++++++++++++++++++++++++++++++++++++++++++++++
		mov	(r1)+,r5
		mov	(r1)+,sp
		mov	(r1),(sp)
		rts	pc
-------------------------------------------------
--	1:
--		mov	4(r1),2(r5)
--		jmp	cret
-------------------------------------------------

Chris Maltby
University of Sydney