Surprising fact about STREAMS service modules

[Larry Philps]

Well, I just learned something about the STREAMS mechanism today.  Since I
found it quite surprising, I thought I should mention it to the world before
some other poor sucker gets surprised also.

As anyone who has perused the STREAMS code at all knows, all the
non-interruptable parts of it run at "splstr", which on most systems is
spl5.  This is lower than the buffer cache, but high enough to block all
devices, most importantly the terminals and the network.  After looking
carefully at the STREAMS programming manuals I can't find any reference to
the spl at which the driver service modules are called.  Silly me, I just
assumed that it would be at splstr.  However, check out the following code
fragment from io/stream.c.

queuerun()
{
	....

	s = splstr();
	...
	if (q->q_qinfo->qi_srvp) {
		spl1();
		(*q->q_qinfo->qi_srvp)(q);
		splstr();
	}
	...
	splx(s);
}

Note the spl1()!  Anybody else out there surprised?  We were seeing some
really bizarre behaviour out of RFS when under heavy load the server would
reverse the order of a DUCOPYOUT and a DUREAD packet.  RFS produces no
diagnostics whatsoever when the out of sequence DUCOPYOUT arrives and the
only visible effect was that part of the read returned all zeros.

We tracked this down using a network analyzer to capture a packet trace and
then analyzed the RFS headers.  After much work I decided that "This can
never happen, but it did - well ... at least it can never happen if the code
is never reentered".  After that it took every little time to discover the
preceeding fragment.

So, the moral of this story is, if you have a STREAMS driver that talks to
any hardware, be sure to protect its code with an splstr() or else write that
code so that it can be reentered.

---
Larry Philps                             HCR Corporation
130 Bloor St. West, 10th floor           Toronto, Ontario.  M5S 1N5
(416) 922-1937                           {utzoo,utcsri,ihnp4}!hcr!larry