How do I keep a pipe from buffering I/O?

[Tom Christiansen]

>From the keyboard of bert at let.rug.nl (Bert Bos):
:I tried to fork a process and redirect both input and output to pipes.
:Both processes could work in parallel and as soon as one of the two
:had some text ready, it would be written out on the pipe for the other
:process to read. One of the processes could even be a program such as
:Awk (started with exec()).
:
:Problem is, the pipes appear to buffer such large amounts of text,
:that one process only gets input after the other has already finished.
:How do I force the pipes to pass on text one line at a time?

You must have control of the program that's doing the output.  If you
don't have that program's source code, you're out of luck.  Many programs
use stdio, and for these you must force a flush as soon as want the output
to show up.  From C, you can use fflush(3S) now and then, or set buffering
to be line- or un-buffered.  See the setbuf(3S) and related functions.
This is how it's usually done.

Now, in the case of awk, you're doubly out of luck: even with the source
code, you aren't going to be able to make it flush its buffers when you
want, because there's no way for awk to do that.  Ok, I take that back:
you run it through the awk-to-perl translator, because in perl it's 
easy to flush your stdio buffers.  In fact, one person posted a short
sh/awk script a few weeks ago with just your problem.   The perl solution
solved his problem.

Ok, I'll take it back again.  If you run your program under a debugger,
you can probably hammer the stdio flags.  Depending on your implementation
you might even be able to patch the binary.  But these are really very last
ditch efforts, and I wouldn't want to admit to having done them. :-)

--tom
--
"UNIX was not designed to stop you from doing stupid things, because
 that would also stop you from doing clever things." -- Doug Gwyn

 Tom Christiansen                tchrist at convex.com      convex!tchrist