4.3 BSD and getting rid of control tty

[Eric Bivona]

I have run across a situation that strikes me as a kernel bug, but
maybe I'm just missing something.  Here's the problem:

I am responsible for a library of code that does (in part) a fork and
exec to run a DBMS as an inferior process.  For various reasons, after
the fork (vfork), the child code runs through the basic disassociate
from control terminal code from the 4.3 BSD Advanced IPC Tutorial.  No
problem yet, everything is fine.

A program that uses this library is often run out of a shell script
that may run for *many* hours, preferably at night.  If the shell
script is started, and the user logs out, the previously mentioned
program hangs in the previously mentioned library, in the disassociate
code, namely:

	if ((fd = open("/dev/tty", O_RDWR | O_NDELAY)) >= 0) {
	    (void) ioctl(fd, TIOCNOTTY, 0);
	    (void) close(fd);
	}

The open call hangs, waiting for someone connect to the unused tty
port.  Since this in the kernel, it is time to get more specific:
4.3 BSD on a VAX 780, and a dz for the ports we're seeing this problem
on (haven't tried on other ports, pty's would be the only other thing
I could easily test this against).  I've looked through the code and I
can see (I think) what isn't happening:  the dz code does nothing
about the O_NDELAY, and the open blocks, until *anyone* logs into the
port that was used by the person running the script.  Maybe somebody
out there is saying: wait, the open is of "/dev/tty", and that's
special, what does it matter with the dz code?  Well, if I follow this
correctly (no, I'm not going to bet on it), the special code used for
/dev/tty is in tty_tty.c, and it just calls the open routine for the
user's tty device, with the tty device from the user (kernel)
structure.

I figure the solutions run something like this:

1) fix the dz code to return EWOULDBLOCK, given O_NDELAY, and an
   unopened tty.  Certainly this would be correct, but it is not
   general: every tty handler would have to be correct (it would be
   nice, but ...).

2) have the logout propagate an empty tty field to the user structure:
   this should occur (I'd guess), or is it overkill of sorts?  I
   haven't dug deep enough to figure this out.

3) change the /dev/tty code to do the 'right' thing if there is
   currently no control terminal. (anyone want to define 'right' as
   used in the previous sentence?  :->).  Might also have to do with
   2, above (can you figure out from the u structure if the control
   terminal has been disconnected?  (there is a comment in this code
   about replacing all of it anyway...).

4) Some other solution, hopefully clean, and in user code space that
   would fix all this?

Thanks in advance!

-Eric Bivona
 DCIS Project
 Dartmouth College
 Eric.J.Bivona at Dartmouth.EDU