tty driver: sleep on output queue

[David McVicar]

We have HP900/800 series running HP-UX 3.1.

I have a problem with two processes that seem deadlocked on a kernel resource.

Background:

Two processes that are trying to access a tty device via the C stream
I/O library:

Process 1 is an end user application process running in the foreground
attempting a fprintf(), and is sleeping in an _xflsbuf().
(The WCHAN, after a little kernel data structure/code examination, showed
the process was sleeping on tp->t_outq for that tty).  

Process 2 is a daemon process whose purpose is to "reap" idle users.  It 
sums CPU usage for a process group attached to a tty, and when that sum falls 
below a certain threshold, the daemon terminates those processes.  Before 
killing anything, the daemon tries to warn the user that death is imminent.  
To do this, the daemon first uses curses setupterm() (loads terminfo), so it 
may read the cursor location for the users TERM type.  It then uses fwrite()
to write a message at the bottom of the screen.

Problem: 

	The daemon process gets hung up on many occasions and is found also
	to be in _xflsbuf() or curses reset_prog_mode() sleeping on the same 
	WCHAN as the foreground application process.  (Two process asleep, 
	waiting on the same event.)

Questions:

	Do you need more background/environmental information ?  I will 
	provide it.

	Are there known problems in UNIX with two processes (one foreground, 
	one background) sleeping on the same event and deadlocked when reading 
	and writing to a tty?

	I have noticed a ioctl(,TCFLSH,) will call wakeup(tp->t_outq).  Would 
	this suffice inside the background process (before attempting output 		to idle terminals) to prevent deadlock?

	If I missed the boat on this one, what else could be causing this?



Thanks for any suggestions.
-- 
mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
    David J. McVicar 		...!uunet!prcrs!mcvic
   PRC Realty Systems 
   McLean, VA   22102