Insufficient Resource Error on msgsnd Call

[Robert J Woodhead]

In article <1023 at dinl.mmc.UUCP> noren at dinl.UUCP (Chuck Noren) writes:
>Process A generates 200 messages in bursts of about 50 as fast as it
>can go (CPU bound) and puts it into Queue 1.  Process B reads the
>messages from Queue 1, processes them while looking things up in
>an Ingres database (we are using Ingres 5.0).  Process B sends even more
>messages to Queue 2 which is read by Process C.
>
>Any suggestions of what could be happening?  Is Ingres using resources
>common to Message Queues?  Have I shown a misunderstanding of how message
>queues are to be used?

What is happening is that there is a certain amount of memory space set
aside for messages in queues (this is a kernel parameter) and you are
filling it up.

Consider:  Your process A is pumping tons of messages into Queue 1, filling
up queue memory space.  Now process B attempts to write to Queue 2, but
there is not enough space to do this, so it halts waiting for space to
become available.  There are only two ways this can happen; if process C
or process B reads messages out of the queues.  B can't do this; it is
halted trying to write to queue 2.  Now, if C, after reading all the
messages in queue 2, still hasn't freed up enough space for B's write
to complete, your pipeline will freeze.  It is even worse, because as
C reads things, process A will probably snarf this space by sending
messages.  Ooops!

The error message you are getting when you specify IPC_NOWAIT in the msgsnd
indicates this is what is happening.  It's occuring because your pipeline
is poorly designed; the majority of the processing is going on in process B,
and tons of messages are piling up in queue 1.

Note that when you use your debug process to read some messages out of the
queues, the blocked write eventually gets done, and the pipeline unblocks.

Solutions:

1) Increase kernel space allocated for messages so that worst case there
   will always be enough room.  Advantage : quick and easy.  Disadvantage :
   Unless you can determine the maximum size needed, it isn't reliable.

2) Add a queue 3 that connects process C to process A.  Process C sends
   a blank message down queue 3 every N messages it gets from queue 2.
   Process 1 has a counter it decrements by 1 each message it sends into
   queue 1, starting at, say 2*N (or M in the general case).  When
   this counter goes to 0, process A reads a message from queue 3 (and
   blocks until it gets one), and then adds N to the counter and
   continues.  This guarantees that there are at most M-N transactions
   in transit, and allows you to ensure that you don't overflow your
   message space.

3) Restructure your system as a client/server model, where B serves A
   and C serves B.  Since either side can put a message in the queue,
   A sends a message to B, then does a msgrcv.  When B reads the message,
   it sends a dummy message back to A, saying ``I got it, you can now put
   another request in''.  B and C do the same process.  End result is that
   there is never more than one message in each queue at any time.

   By using two queues between each process, you can generalize this to
   never more than N message in each queue at any time.

-- 
Robert J Woodhead, Biar Games, Inc.  !uunet!biar!trebor | trebor at biar.UUCP
"The lamb will lie down with the lion, but the lamb won't get much sleep."
     -- Woody Allen.