Weird code

[Mike Laman]

In article <468 at isucs1.UUCP> justice at isucs1.UUCP writes:
>
>Ok you wizards out there, why does this program do what it does?
>Here are a couple of hints:
>	1) It only works on Vaxes
	:
>main()
	:
>	int func();
>
>	fix((int *) func);
	:
>fix(f)
>int *f;
>{
>        *f &= ~0xc00;
>}
	:

Someone else will probably answer this before me -- we always seem to lag
behind on the news.  I apologize if someone else has answered it "already".

The routine "fix" is modifying a mask that tells the "calls" (and friends)
instruction which registers (if any) to save ON A VAX 11/780 (and friends).
The mask is kept in the "first word" of the function which is where "func"
points.  "Fix" is changing the mask so registers 11 and 10 will NOT be
saved/restored across function calls.  On the VAX the UNIX C compiler allocates
register 11 for the first register variable, then decreasing number in order of
allocation of registers.  This means the first two register variables
(if the compiler see fit to use them as registers -- that's another
story) will get overwritten in the calling routine ("main") every time it
calls the routine that was "fixed", namely "func".  In other words "func"
is modifying "main"'s register variables.

And I thought all that assembly I played with a year ago would be just
a useless effort in futility :-).

		Mike Laman, NCR Rancho Bernardo
		UUCP: {ucbvax,philabs,sdcsla}!sdcsvax!ncr-sd!laman