How do *you* debug device drivers?

[Rob Warnock]

+---------------
| [Me:] As I already sent a copy to the original requestor, I will not post it
| here (~200 lines) unless the demand warrents...
+---------------

Nine requests in six days... o.k., here it is (attached below). This was only
an extended abstract, and in any case did not advocate a specific "system",
but was more of a general approach to the problem.

Notes:	1. This material is Copyright 1984 by USENIX Association,
	   as it appeared in the Proceedings, posted with permission.
	   Any use other than for personal education must be authorized
	   in writing by USENIX.

	2. Use "nroff -ms".
	
	3. The work described was done for Fortune Systems, a previous employer.
	   Requests for further details or code should be addressed to them.


Rob Warnock
Systems Architecture Consultant

UUCP:	{ihnp4,ucbvax!dual}!fortune!redwood!rpw3
DDD:	(415)572-2607
USPS:	510 Trinidad Lane, Foster City, CA  94404


-----cut here---------cut here---------cut here------------cut here-----
.RT
.nr LL 7i
.ll 7i
.nr PO 0.75i
.po 0.75i
.DA 20 May 1984
.TL
User-Mode Development
Of Hardware and Kernel Software
.AU
Robert P. Warnock, III
.AI
Fortune Systems Corporation
Redwood City, California 94061
.AB
As a general rule, the development of new hardware devices,
operating systems drivers for those devices,
and other new operating systems functions
is considerably more difficult than the development of
user-mode functions of similar complexity.
Several factors contribute to this:
hardware often doesn't work as initially expected (despite documentation);
testing drivers and other kernel functions requires a very scarce resource \(em
standalone time on the system;
errors often leave the entire system hung or halted with no history trace,
making crash analysis a challenge at best;
the edit-compile-load cycle tends to be longer and more complex;
and a logic analyzer is seldom the most convenient diagnostic tool.

A set of techniques or "tricks" are presented,
with examples of their application.
While each one may be "obvious" by itself,
and not particularly related to the others,
together they illustrate a common principle and general method.
The principle is that of separation of concerns,
together with addressing those concerns in the proper order.
"First make it work correctly;
then make it work well while remaining correct."
The general method is to do the development in user-mode software,
using minimal "hooks" to make this possible.
Then, after the functionality has been demonstrated and the critical
algorithms debugged,
the software is "ported" to kernel mode as necessary to
attain the required performance goals.

Other authors [Holt] [Wulf]
have suggested, in fact, that the
"kernel" of an operating system should be quite tiny
(a few hundred lines of assembler),
and that ALL of what one normally thinks of as the "operating system"
should be run in user-mode, including device drivers, file systems,
and schedulers.
Unfortunately, most of us do not have the freedom to make major
modifications to our operating system environment (typically
.UX
of some flavor or other).
The examples given demonstrate that, at least during initial development,
it is possible to obtain the benefits of the "user-mode style"
even though the production version may be completely traditional in structure.

The development projects used as examples took place at Fortune Systems
between Summer 1982 and Summer 1984, and include:
.IP 1.
A byte-parallel file-transfer link was implemented between a DEC VAX-11/780
and a Fortune Systems 32:16.
The VAX driver was developed in user mode using /dev/kUmem
to access the hardware.
The 32:16 driver was developed in user mode using the "sysphys" feature
(UNIX Edition 7 "phys(2)" call) to map the user addresses to the hardware.
After the file-transfer application was completely functional, the
VAX driver was moved to the kernel, with a 25-fold improvement in performance.
(The 32:16 driver was left in user-mode permanently.)
.IP 2.
A communications co-processor for the 32:16 was debugged using user-mode
software (again using "sysphys").
When the UNIX driver was being debugged, host-resident user-mode code
was used to mimic the co-processor application on the one hand,
while making calls to the driver and comparing the results on the other.
A similar procedure was used in developing a bit-mapped
graphics controller and a parallel-I/O co-processor.
.IP 3.
A set of library subroutines was written to allow user-mode emulation
of (proposed) new operating system calls.
When the "system call" was invoked, instead of entering the normal
(kernel-mode) system call handler,
a call-request packet was passed through a "pty" to a daemon program which
emulated the call and passed a "return value" packet back through the pty.
Packet types were provided to allow the daemon to read and write
the client process's address space (as the kernel would have been able to do).

This facility was used to develop a network "socket" mechanism
(similar to 4.2bsd sockets).
A "network line discipline" was implemented
using ordinary terminal ports as network devices.
After the internet router and network line discipline were completely
functional running in user mode as a system-call emulation daemons
(including actually transmitting packets over a multi-host net),
they were "ported" straightforwardly into the kernel.
.IP 4.
In the previous hardware examples, the physical device had its interrupts
disabled when driven by the user-mode driver, so as not to crash the
unmodified naive kernel with unexpected interrupts.
(The user-mode drivers used either busywait-polling or sleep-polling
for synchronization.)
Similarly, DMA operation was not possible.

In developing a local-area network interface, it was necessary to
utilize both of those features.
A slight kernel modification was made to reserve a block of physical
memory which the kernel would not use.
User-mode library routines were provided that
(1)\ allowed allocation of that memory area to DMA operations
(the results of which were then examined with "/dev/mem" or "sysphys"),
and (2) allowed run-time installation of minimal interrupt-service routines
(using "pre-compiled" templates) which merely stored the device status
in a mailbox and cleared the interrupt
(the user-mode driver polled the mailbox, rather than the hardware).

Again, the device driver was not "ported" to kernel mode until
the hardware had been completely checked out,
the device driver algorithms were debugged,
and the sample application programs had demonstrated end-to-end functionality.
.LP
Several examples have been given of developing what is normally considered
"kernel mode" software in user mode.
While these examples are not likely to apply directly to other environments,
it is hoped that implementors will be encouraged to consider the
"user-mode style" when planning future kernel-mode software
development projects.
.AE
.FS
.sp 1
[Holt]
R. C. Holt,
.I
Concurrent Euclid, The UNIX System, and Tunis,
.R
Addison-Wesley, 1983
.FE
.FS
.sp 1
[Wulf]
William A. Wulf, Roy Levin, and Samuel P. Harbison,
.I
HYDRA/C.mmp,
.R
McGraw-Hill, 1981
.FE