dup()'ed pipe()'s to stdio

[Daniel E. Platt]

To make a long story short:

I have a program that wants to FTP a slew of files from a box with
lots of disk space to a box with small disk space on demand.  Since
I have LOTS of files, I only want this program to invoke FTP once,
and then the parent can block on a read of 'ftp's output to see
when the file was transfered, rather than breaking the network
connection, and re-invoking 'ftp' everytime I wanted another file.

With this in mind:
I want to open two pipes between a parent and a child; the parent has
opened one pipe dup()'ed in the child process to stdin, and the other
pipe opened by the parent was dup()'ed in the child to stdout.  The
parent writes to the pipe duped to the child's stdin, and reads from
the pipe dup()'ed to the child's stdout.  The child then exec's 'ftp'.
'Ftp's stdout is buffered.  When it detects that it isn't attached to
a tty device, it acts like its buffering to disk, and never gets around
to writing the short responses back to the parent.  BESIDES putting
'setbuf()'s or 'fflush()'s into 'ftp' -- a program which I don't have
the code to anyway -- how can I get 'ftp' to send its responses
unbuffered?

In article <22266 at mimsy.umd.edu>, chris at mimsy.umd.edu (Chris Torek) writes:
> This has little to do with C per se, and belongs in comp.unix.questions.

I receive a partial news feed at my home system.  I don't have lots of
resources to clutter up my system with; this was the most germain
group of those I subscribe to.  Sorry... I'm really not trying to
screw things up for other people...

> 
> In article <273 at ndla.UUCP> platt at ndla.UUCP (Daniel E. Platt) writes:
> >I have a question about what happens to the buffering of stdio
> >when stdio is re-directed via a dup() from a pipe()'ed file
> >descriptor before exec'ing.
> 
> exec() throws away the current program, *including all information
> stdio has ever built up*.  (Some people attempt to shut off buffering
> by adding a `setbuf(stdout, (char *)NULL)' call before the exec.  This
> cannot work, because exec() throws out that information.)

I know that 'exec' throws out all buffer stuff.  I've not lost buffer
stuff.  The problem is that I'm trying to set up two way communications
between a parent and 'ftp', and 'ftp's stdout is buffered.  Besides,
I have not re-directed stdio in the parent process, only in the child
process.  Also, all I have in the parent before fdopen() is file
descriptors returned by pipe().  There's no buffer to set to zero
in this communication stream.

> 
> >In the [child program with stdin & stdout being a pipe], if I have
> >something like:
> >
> >	while(scanf("%d", &i) == 1)
> >		printf("%d", i * i);
> >
> >without first calling:
> >	
> >	setbuf(stdin, NULL);
> >	setbuf(stdout, NULL);
> 
> (Unless you have a machine that uses function prototypes, these should
> be `setbuf(stdin, (char *)NULL)' and `setbuf(stdout, (char *)NULL)'.)
> 
> >what happens is the parent hangs.  It would appear that the child won't
> >write its buffer until it fills it up... just like it was writing to
> >a disk.
> 
> In some versions of Unix, a pipe *is* a disk file.  (In others it is a
> `stream' or a `socket'; but in most if not all Unix systems, a pipe is
> not a `tty' device.)  Stdio believe in buffering: with the exception of
> `tty' devices, all output is buffered in large chunks.  On `tty's
> (whatever those are: in 4BSD, a tty is a device that allows
> ioctl(TIOCGETP)), stdio buffers output only to a newline (or a `large
> chunk', whichever comes first).

This is the problem; I am trying to invoke a program with pipes installed
both to stdin and stdout (in this, I've succeeded).  However, this program
has used buffered stdio routines for VERY low volume screen IO.  What
happens is the buffering blocks my read() in the parent.

> 
> >However, with the setbuf()'s present, there is no hangup.  I assume
> >that it knows to create the buffer when it determines that it has been
> >re-directed.
> 
> The child is a completely new program.  The first time it tries to
> do something with stdio, it decides that it should buffer stdout.  If
> stdout is a `tty', stdio buffers it up to newlines; otherwise stdio
> buffers it fully.

I know about this; it's in the man pages about stdio.  Again, I was
hoping that somebody would have a bright idea about how to unblock
buffers; perhaps to get it to think it was still attached to a tty?

> 
> >However, doesn't it know that it was redirected from
> >a pipe as opposed to being re-directed to or from a disk file?
> 
> Stdio cares not in the least about anything except `tty' devices.

This is sad; it makes it difficult to call 'ftp' from other programs;
or actually lots of other programs too.

> 
> >If I'm trying to do this to a program for which I only have the binary,
> >and which uses stdio buffered, is there a way to fool it into not using
> >a buffer?
> 
> Such a program is, as far I am concerned, buggy.  Programs that
> interact, but depend on `tty' style buffering, are in need of repair.
> ANY program that intends to interact---whether with a person or another
> program---should, if it uses stdio, use fflush() before each input
> request to make sure that its output goes out.  I think it was a
> mistake to add line-buffering to stdio at all: convenient perhaps, but
> a mistake, for it voids the model `everything is a byte stream file'.

I'm glad that you feel about this the same way I do.  From your
description, you're not saying the parent program is incorrect in
concept or in execution, but that the program that the parent
is invoking, namely 'ftp', is buggy (I'm asserting this since this
program was a *Program that interacts, but depends on `tty' style
buffering*).  From my standpoint, it has rendered 'ftp' unusable
as a solution to my problem.

> 
> Programs that cheat---that call setbuf to disable all buffering---are
> not so broken, but are performance disasters.  The right solution is to
> put fflush calls into the offending program.  Of course, this is sometimes
> not possible.

I don't have the source to 'ftp' to put in the 'fflush()'s.  The volume
of information that I'm passing between parent and child isn't that
great however, so I'm NEVER going to come close to 1K before I need
to flush the buffer in any event.

> 
> If all else fails, the last trick is to run the program after
> connecting its stdout to what the program will think *is* a `tty'.
> Most modern Unix systems have `pseudo terminals' that can be used for
> this purpose...

I think I may try it.  Unfortunately, this is VERY system specific.  I
believe they are called `pty00' - `ptyxx' on the systems at work,
but on my home system, they're called 'pts0', etc.

By the way, I haven't tried messing with this sort of kludge much before.
I assume this type of device is essentially 'special' file, and that it
acts sort of like a FIFO?

>                ...In extreme situations, you can connect a real terminal
> port back to a second terminal port, and make the machine talk to
> itself over a serial line.  (This particular kludge is something you
> would expect of an IBM O/S, not of Unix.)

This is gross and disgusting :-(  Besides, I don't want to connect
cables everytime I want to run my application on a new system.  I'd
run out of cables.

> -- 
> In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163)
> Domain:	chris at cs.umd.edu	Path:	uunet!mimsy!chris

I appreciate the information in this article; it pretty much confirms
what I thought (I didn't mess up on my homework).  This is very
unfortunate however... I was hoping that I HAD screwed up, and that there
WAS an easy way around my problem.  The `pseudoterminals' sounds like the
best option to solve this problem in the short run.

Dan
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
||                                     1(914)945-1173            ||
||        Dan Platt                    1(914)941-2474            ||
||        Watson (IBM)                 PLATT at YKTVMV.BITNET       ||
||                           ..!uunet!bywater!scifi!ndla!platt   ||
||                                                               ||
||     The opinions expressed here do not necessarily reflect    ||
||                   those of my employer!                       ||
||                                                               ||
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-