Operating Systems B.

RANDELL, Editor

The Nucleus of a In the following, the philosophy and structure of the

R C 4000 multiprogramming system is explained. The dis-
Multiprogramming System cussion does not include details of implementation; size
and performance are presented, however, to give an idea of
the feasibility of this approach. The functional specifica-
PIER BRINCIt HANSEN tions of the multiprogramming system are described in
A / S Regnecentralen, Copenhagen, Denmark detail in a report [1] available from Regnecentralen.

2. System Nucleus
This p a p e r describes the philosophy and structure of a multi- Our basic attitude during the designing was to make no
programming system that can be extended with a hierarchy of assumptions about the particular strategy needed to
operating systems to suit diverse requirements of program optimize a given type of installation, but to concentrate on
scheduling and resource allocation. The system nucleus sim- the fundamental aspects of the control of an environment
ulates an environment in which program execution and input/ consisting of parallel, cooperating processes.
output are handled uniformly as parallel, cooperating proc- Our first task was to assign a precise meaning to the
esses. A fundamental set of primitives allows the dynamic process concept, i.e. to introduce an unambiguous ter-
creation and control of a hierarchy of processes as well as the minology defining what a process is and how it is imple-
communication among them. mented on the actual computer.
KEY WORDS AND PHRASES: multiprogramming,operating systems, paralle I The next step was to select primitives for the synchro-
processes, processconcept, processcommunication,message buffering, process nization and transfer of information among parallel
hierarchy, processcreation, process removal processes.
CR CATEGORIES: 4.30, 4.31, 4.32, 4.41 Our final decisions concerned the rules for the dynamic
creation, control, and removal of processes.
The purpose of the system nucleus is to implement these
l, Introduction fundamental concepts: simulation of processes; communi-
The multiprogramming system developed by Regnecen- cation among processes; creation, control, and removal of
tralen for the RC 4000 computer is a general tool for the processes.
design of operating systems. I t allows the dynamic creation
of a hierarchy of processes in which diverse strategies of 3. Processes
program scheduling and resource allocation can be imple- We distinguish between internal and external processes,
mented. roughly corresponding to program execution and i n p u t /
For the designer of advanced information systems, a. output.
vital requirement of any operating system is t h a t it allow More precisely, an internal process is the execution of one
him to change the mode of operation it controls; otherwise. or more interruptable programs in a given storage area. An
his freedom of design can be seriously limited. Unfortu- internal process is identified b y a unique process name.
nately, this is precisely what present operating systems do, Thus other processes need not be aware of the actual loca-
not allow. Most of them are based exclusively on a single, tion of an internal process in the store, but can refer to it by
mode of operation, such as batch processing, priority name.
scheduling, real-time scheduling, or conversational access. A sharp distinction is made between the concepts pro-
When the need arises, the user often finds it hopeless to, gram and internal process. A program is a collection of
modify an operating system that has made rigid assump- instructions describing a computational process, whereas
tions in its basic design about a specific mode of operation. an internal process is the execution of these instructions in
The alternative--to replace the original operating system a given storage area.
with a new one--is in most computers a serious, if not im- In connection with i n p u t / o u t p u t , the system distin-
possible, matter because the rest of the software is inti- guishes between peripheral devices, documents, and ex-
mately bound to the conventions required by the origina~ ternal processes.
system. A peripheral device is an item of hardware connected to
This unfortunate situation indicates that the mairr. the data channel and identified by a device number. A
problem in the design of a multiprogiramming system is not. document is a collection of data stored on a physical
to define functions t h a t satisfy specJ.fie operating needs, but~ medium, such as a deck of punched cards, a printer form, a
rather to supply a system lmeleus that can be extended: reel of magnetic tape, or a file on the backing store.
with new operating systems i~ an orderly manner. This is: An external process is the i n p u t / o u t p u t of a given docu-
the primary objective of the R C 4000 system. ment identified by a unique process name. This concept

238 Comxnuuieatton~ ~f the~ACM Volume 13 / Number 4 / April, 1970

implies t h a t internal processes can refer to documents by if it is waiting for the answer. T h e answering process con-
name without knowing the actual devices on which they tinues immediately.
are mounted. W a i t answer delays the requesting process until an
Multiprogramming and communication between inter- answer arrives in a given buffer. On arrival, the answer is
nal and external processes are coordinated b y the system copied into the process and the buffer is returned to the
nucleus--an interrupt response program with complete pool. Th6 result specifies whether the answer is a response
control of i n p u t / o u t p u t , storage protection, and the inter- from another process or a d u m m y answer generated b y the
rupt system. We do not regard the system nucleus as an system nucleus in response to a message addressed to a
independent process, but rather as a software extension of nonexisting process.
the hardware structure, which makes the computer more T h e procedure wait message forces a process to serve its
attractive for mu]tiprogramming. I t s function is to imple- queue on a first-come, first-served basis. The system, how-
m e n t our process concept and primitives t h a t processes can ever, also includes two primitives t h a t enable a process to
invoke to create and control other processes and communi- wait for the arrival of the next message or answer and serve
cate with them. its queue in any order.
So far we have described the multiprogramming system This communication scheme has the following advan-
as a set of independent, parallel processes identified b y tages.
names. The emphasis has been on a clear understanding of The multiprogramming system is dynamic in the sense
relationships among resources (store and peripherals), data t h a t processes can appear and disappear at any time.
(programs and documents), and processes (internal and Therefore a process does not in general have a complete
external). knowledge of the existence of other processes. This is
reflected in the procedure wait message, which makes it
4. Process Communication possible for a process to be unaware of the existence of
I n a system of parallel, cooperating processes, mecha- other processes until it receives messages from them.
nisms must be provided for the synchronization of two On the other hand, once a communication has been
processes during a transfer of information. established between two processes (i.e. by means of a
Dijkstra has demonstrated t h a t indivisible lock and message) they need a common identification of it in order
unlock operations operating on binary semaphores are to agree on when it is terminated (i.e. b y means of an
sufficient primitives from a logical point of view [3]. We answer). T h u s we can properly regard the selection of a
have been forced to conclude, however, t h a t the semaphore buffer as the creation of an identification of a conversation.
concept alone does not fulfill our requirements of safety A h a p p y consequence of this is t h a t it enables two processes
and efficiency in a dynamic environment in which some to exchange more t h a n one message at a time.
processes m a y turn out to be black sheep and break the We m u s t be prepared for the occurrence of erroneous or
rules of the game. malicious processes in the system (e.g. undebugged pro-
Instead we have introduced message buffering within the grams). This is tolerable only if the system nucleus
system nucleus as the basic means of process communica- ensures t h a t no process can interfere with a conversation
tion. The system nucleus administers a common pool of between two other processes. This is done b y storing the
message buffers and a message queue for each process. identity of the sender and receiver in each buffer and check-
The following primitives are available for the communi- ing it whenever a process a t t e m p t s to send or wait for an
tion between internal processes: answer in a given buffer.
send message (receiver, message, buffer), Efficiency is obtained b y the queueing of buffers, which
wait message (sender, message, buffer), enables a sending process to continue immediately after
send answer (result, answer, buffer), delivery of a message or an answer, regardless of whether
wait answer (result, answer, buffer). or not the receiver is ready to process it.
Send message copies a message into the first available To make the system dynamic, it is vital t h a t a process
buffer within the pool and delivers it in the queue of a can be removed at any time, even if it is engaged in one or
named receiver. The receiver is activated if it is waiting for more conversations. I n this case, the system nucleus leaves
a message. T h e sender continues after being informed of all messages from the removed process undisturbed in the
the identity of the message buffer. queues of other processes. When these processes answer
W a i t message delays the requesting process until a mes- them, the system nucleus returns the buffers to the com-
sage arrives in its queue. When the process is allowed to mon pool.
proceed, it is supplied with the name of the sender, the The reverse situation is also possible: during the removal
contents of the message, and the identity of the message of a process, the system nucleus finds unanswered messages
buffer. T h e buffer is removed from the queue and made sent to the process. These are returned as d u m m y answers
ready to transmit an answer. to the senders.
Send answer copies an answer into a buffer in which a T h e main drawback of message buffering is t h a t it intro-
message has been received and delivers it in the queue of duces yet another resource problem, since the common
the original sender. T h e sender of the message is activated pool contains a finite number of buffers. If a process were

V o l u m e 13 / N u m b e r 4 / A p r i l , 1970 Communications o f t h e ACM 239

allowed to empty the pool by sending messages to igno- 6. Internal Processes
rant processes, which do not respond with answers, further
A final set of primitives in the system nucleus allows the
communication within the system would be blocked. Con-
creation, control, and removal of internal processes.
sequently a limit is set to the number of messages a process
Internal processes are created on request from other
can send simultaneously. B y doing this, and by allowing a
internal processes. Creation involves the assignment of a
process to transmit an answer in a received buffer, we have
name to a contiguous storage area selected by the parent
placed the entire risk of a conversation on the process that
process. The storage area must be within the parent's own
opens it. area.
After creation, the parent process can load a program
5. External Processes
into the child process and start it. The child process now
Originally the communication primitives were designed shares computing time with other active processes includ-
for the exchange of messages between internal processes. ing the parent process.
Later we also decided to use send message and wait answer On request from a parent process, the system nucleus
for communication between internal and external processes. waits for the completion of all i n p u t / o u t p u t initiated by a
For each kind of external process, the system nucleus child process and stops it. I n the stopped state, the process
contains a piece of code that interprets a message from an can still receive messages and answers in its queue. These
internal process and initiates i n p u t / o u t p u t using a storage can be served when the process is restarted.
area specified in the message. When i n p u t / o u t p u t is termi- Finally, a parent process can remove a child process in
nated by an interrupt, the nucleus generates an answer to order to assign its storage area to other processes.
the internal process with information about actual block According to our philosophy, processes should have
size and possible error conditions. This is essentially the complete freedom to choose their own strategy of program
implementation of the external process concept. scheduling. T h e system nucleus only supplies the essential
We consider it to be an important aspect of the system primitives for initiation and control of processes. Conse-
t h a t internal and external processes are handled uniformly quently, the concepts of program loading and swapping are
as independent, self-contained processes. The difference not part of the nucleus. Time-sharing of a common storage
between them is merely a matter of processing capability. area among child processes on a swapping basis is possible,
A consequence of this is that any external process can be however, because the system does not check whether inter-
replaced by an internal process of the same name if more nal processes overlap each other as long as they remain
complex criteria of access and response become desirable. within the storage areas of their parents. Swapping from
External processes are created on request from internal process A to process B can be implemented in a parent
processes. Creation is simply the assignment of a name to a process as follows: stop(A) ; output(A); input(B) ; start(B).
particular peripheral device. To guarantee internal proc-
esses exclusive access to sequential documents, primitives 7. Process Hierarchy
are available for the reservation and release of external T h e idea of the system nucleus has been described as the
processes. simulation of an environment in which program execution
Typewriter consoles are the only external processes that and i n p u t / o u t p u t are handled uniformly as parallel, co-
can send messages to internal processes. T h e operator operating processes. A fundamental set of primitives allows
opens a conversation by pushing an interrupt key and the dynamic creation and control of processes as well as
typing the name of the internal receiver followed by a line communication among them.
of text. For a given installation we still need, as part of the sys-
A file on the backing store can be used as an external tem, programs that control strategies of operator com-
process by copying a description of the file from a catalog munication, program scheduling, and resource allocation;
on the backing store into the system nucleus; following but it is essential for the orderly growth of the system that
this, internal processes can initiate i n p u t / o u t p u t by send- these operating systems be implemented as other programs.
ing messages to the file process. Since the difference between operating systems and pro-
Real-time synchronization of internal processes is ob- duction programs is one of jurisdiction only, this problem
tained by sending messages to a clock process. After the is solved by arranging the internal processes in a hierarchy
elapse of a time interval specified in the message, the clock in which parent processes have complete control over child
returns an answer to the sending process. processes.
I n general, external processes can be used to obtain After initial loading, the internal store contains the sys-
synchronization between internal processes and any signal tem nucleus and a basic operating system, S, which can
from the external world. For example, an internal process create parallel processes, A, B, C, etc., on request from
may send a message to a watchdog process and receive an consoles. T h e processes can in turn create other processes,
answer when a magnetic tape is mounted on a station. In D, E, F, etc. Thus while S acts as a primitive operating
response, the internal process can give the station a tem- system for A, B, and C, these in turn act as operating sys-
porary name, identify the tape by reading its label, and tems for their children, D, E, and F. This is illustrated by
rename the station accordingly. Figure 1, which shows a family tree of processes on the left

240 Communications of the ACM Volume 13 / Number 4 / April, 1970

and the corresponding storage allocation on the right. This 8. Implementation
family tree of processes can be extended to any level, sub- The R C 4000 is a 24-bit, binary computer with typical
ject only to a limitation of the total number of processes. instruction execution times of 4 microseconds [2]. I t per-
In this multiprogramming system, all privileged func- mits practically unlimited expansion of the internal store
tions are implemented in the system nucleus, which has no and standardized connection of all kinds of peripherals.
built-in strategy. Strategies can be introduced at the var- Multiprogramming is facilitated by program interruption,
ious higher levels, where each process has the power to storage protection, and privileged instructions.
control the scheduling and resource allocation of its The present implementation of the system makes multi-
children. The only rules enforced by the nucleus are the programming feasible with a minimum store of 16K-32K
following: a process can only allocate a subset of its own words backed b y a fast drum or disk. The system nucleus
resources (including storage and message buffers) to its includes external processes for a real-time clock, type-
children; a process can only start, stop, and remove its own writers, paper tape input/output, line printer, magnetic
children (including their descendants). After removal of a tape, and files on the backing store. The size of the nucleus
process, its resources are returned to the parent process. and the basic operating system is as follows:
words
SYSTEM NUCLEUS primitives 2400
code for external processes 1150
process descriptions and buffers 1250
A D
E system nucleus 4800
S basic operating system 1400

B
F G 6200
H
The communication primitives are executed in the un-
interruptable mode within the system nucleus. The execu-
tion times of these set a limit to the system's response to
Fro. 1 real-time events:
msec
Initially all system resources are owned by the basic send message 0.6
operating system S. For details of process control and re- wait answer 0.4
source allocation, the reader should consult the manual of wait message 0.4
the system [1]. send answer 0.6
We emphasize that the only function of the family tree
is to define the rules of process control and resource alloca- An analysis shows t h a t the 2 milliseconds required by a
tion. Computing time is shared by round-robin scheduling complete conversation (the sum of the four primitives) are
among active processes regardless of their position in the used as follows:
hierarchy, and each process can communicate with all percent
other processes. validity checking 25
Regarding the future development of operating systems, process activation 45
the most important characteristics of the system can now message buffering 30
be seen as the following.
This distribution is so even that one cannot hope to in-
1. New operating systems can be implemented as other
crease the speed of the system by introducing additional,
programs without modification of the system nucleus. In
ad hoc machine instructions. The only realistic solution is
this connection, we should mention that the ALGOL and
to make the hardware faster.
FORTRAN languages for the RC 4000 contain facilities for
T h e primitives for creation, start, stop, and removal of
calling the nucleus and initiating parallel processes. Thus
processes are implemented in an anonymous internal
it is possible to write operating systems I in high-level lan-
process within the system nucleus to avoid intolerably long
guages.
periods in the uninterruptable mode. Typical execution
2. Operating systems can be replaced dynamically, thus
times for these are:
enabling an installation to switch among various modes of
operation; several operating systems can, in fact, be active msec
simultaneously. create process 3
3. Standard programs and user programs can be start process 26
executed under different operating systems without modi- stop process 4
fication, provided there is common agreement on the possi- remove process 30
ble communication between parents and children. (Continued on page 250)

Volume 13 / Number 4 / April, 1970 Communications of the ACM 241

The analysis presented here suggests that spatial domains 9. ARMOUR, GORDON C., AND BUFFA, Elwoov. A heuristic
are the primitive element of this particular graphic algorithm and simulation approach to relative location of
facilities. Man. Sci. (Jan. 1963), 244-309.
language. In this light, the common assumption that line 10. LEE, R. B. AND MOORE, J. M. CORELAP--computerized
segments are the primitives of many graphic languages relationship layout planning, J. Indust. Eng., 18, 3 (Mar.
may require revision. 1967) 195-200.
11. SIMPSON,M. G., ET AL. The planning of multi-storybuildings:
RECEIVED JUNE, 1969; REVISED OCTOBER, 1969 a systems analysis and simulation approach. Proc. European
REFERENCES Meeting on Statistics, Econometrics and Management
1. GRoss, MAURICE, AND NIVAT, MAURICE. A command Science, Amsterdam, Sept. 1968.
language for visualization and articulated movements. In 12. BARKEN, ROBERT. A set of algorithms for automatically
Computer and Information Sciences II, Julius T. Tou (Ed), laying out hybrid integrated circuits. Internal working doc.,
Academic Press, New York, 1967. Bell Telephone Lab., Holmdel, N. J., Aug. 1968.
2. NILSSON, NILS J. A mobile automaton: An application of 13. NILSSON, N. J., AND RAPHAEL, B. Preliminary design of an
artificial intelligence techniques. Proc. Int. Joint Conf. intelligent robot. In Computer and Information Sciences II,
Artificial Intelligence, May 1969, Washington, D. C. Julius T. Tou (Ed.), Academic Press, New York, 1967.
3. EASTMAN, CHARLES M. Explorations of the cognitive proc- 14. ROSEN, C. A., AND NILSSON, N. J. Application of intelligent
esses of design, Dep. of Comput. Sci., Carnegie-Mellon U., automata to reconnaisance. SRI Project 5953, Third Interim
Feb. 1968, ARPA Rep. DDC No. AD671158, Clearinghouse, Report, Rome Air Develop. Center, Rome, N. Y., Dec. 1967.
Springfield, VA 22151. 15. FAIR, G. R., FLOWERDEW, ET AL. Note on the computer as
4. EASTMAN, CHARLES M. Cognitive processes and ill-defined an aid to the architect. Comput. J. 9, 1 (June 1966).
problems: A case study from design, Proc. Int. Joint Conf. 16. GRISWOLD,R., POAGE, J., AND POLONSKY, I. The SNOBOIA
Artificial Intelligence, May 1969, Washington, D. C. programming language. Bell Telephone Lab., Holmdel,
5. HOWDEN, W. E. The sofa problem. Comput. 3". 11, 3 (Nov. N. J., Aug., 1968.
19687, 299-301. 17. McCARTHY, JOHN, ET AL. LISP1.5 Programmer's Manual.
6. SUTHERLAND, I. E. Sketchpad: a man-machine graphical M I T Press, Cambridge, Mass., 1965.
communication system. Prec. AFIPS 1963 Spring Joint 18. MORAN, THOMAS. Structuring three-dimensional space for
Comput. Conf., Vol. 23, Spartan Books, New York, pp. 329- computer manipulation. Dep. Comput. Sci. working
346. paper, Carnegie-Mellon U., Pittsburgh, Pa., June, 1968.
7. GRAY, J. C. Compound data structure for computer aided 19. MORAN, THOMAS. A model of a multi-lingual designer. In
design: a survey, Proc. ACM 22nd Nat. Conf. 1967, Thomp- Emerging Methods in Environmental Design and Planning,
son Book Co., Washington, D. C., pp. 355--365. G. Moore (Ed.), M I T Press, Cambridge, Mass. (in press).
8. THOMAS, E. M. GRASP--~ graphic service program. Proc. 20. WYLIE, C. ROMNEY, ET AL. Halftone perspective drawings by
ACM 22nd Nat. Conf., 1967, M D I Publications, Wayne, Pa., computer. Teeh. Rep. 4-2, Comput. Sci. Dep., U. of Utah,
pp. 395-402. Salt Lake City, Utah, Feb. 1968.

Hansen--cont'd from page 241 Acknowledgments. The design philosophy was de-
veloped by J~rn 5ensen, S~ren Lauesen, and the author.
The excessive times for the start and removal of an internal Leif Svalgaard participated in the implementation and
process are due to the peculiar storage protection system of testing of the final product.
the RC 4000, which requires the setting of a protection key Regarding fundamentals, we have benefited greatly from
in every storage word of a process. Dijkstra's analysis of cooperating sequential processes.
RECEIVED JULY, 1969; REVISED JANUARY, 1970
9. Conclusion
REFERENCES
Ideas similar to those described here have been sug- 1. RC $000 Soflware: Multiprogramming System. P. Brinch Hansen
gested by others [4-6]. We have presented our system (Ed.). A/S Regnecentralen, Copenhagen, 1969.
because we feel that, taken as a whole, it represents a sys- 2. RC 4000 Computer: Reference Manual. P. Brinch Hansen (Ed.).
tematic and practical approach to the design of replaceable A/S Regnecentralen, Copenhagen, 1969.
operating systems. As an inspiration to other designers, it 3. DIJKSTRA, E. W. Cooperating Sequential Processes. Math.
Dep., Technological U., Eindhoven, Sept. 1965.
is perhaps most important that it illustrates a sequence of 4. I'IARRISON, M. C., A N D SCHWARTZ, J. W. SHARER, a time
design steps leading to a general system nucleus, namely, sharing system for the CDC 6600. Comm. ACM 10, (Oct. 1967),
the definition of the process concept, the communication 659.
scheme, and the dynamic creation and structuring of 5. I'IUXTABLE, D. H. R., AND WARWICK, M. T. Dynamic super-
processes. visors--their design and construction. Proc. ACM Syrup. on
Operating System Principles, Gatlinburg, Tenn., Oct. 1--4,
We realize, of course, that a final evaluation of the sys- 1967.
tem can only be made after it has been used to design a 6. WICHMANN, B. A. A modular operating system. Proc. I F I P
number of operating systems. Cong. 1968, North Holland Pub. Co., Amsterdam, p. C48.

250 C o m m u n i c a t i o n s o f t h e ACM V o l u m e 13 / N u m b e r 4 / Apri|,r1970