Distributed Operating System

(DOS)

1
 Topic
• Introduction
• Preliminary/Recap on important information
– H/w concepts
– S/w concepts
– Middleware
• The operating system layer
• Processes and threads
• Communication and invocation
• Operating system architecture
 Introduction
• Before we even start talking about
Distributed OS – see all related concepts
• To understand better

3
 Hardware Concepts

1.6

Different basic organizations and memories in distributed

computer systems
 Multiprocessors (1)
• A bus-based multiprocessor .
 Multiprocessors (2)

1.8

a) A crossbar switch
b) An omega switching network
Homogeneous Multicomputer
Systems

1-9

a) Grid
b) Hypercube
 Software Concepts
System Description Main Goal

Tightly-coupled operating system for multi- Hide and manage

DOS
processors and homogeneous multicomputers hardware resources

Loosely-coupled operating system for Offer local services to

NOS
heterogeneous multicomputers (LAN and WAN) remote clients

Additional layer atop of NOS implementing Provide distribution

Middleware
general-purpose services transparency
 Uniprocessor Operating Systems

1.11

a microkernel (also known as

μ-kernel) is the near-minimum
amount of software that can
Separating applications from operating system provide the mechanisms
code through a microkernel. needed to implement an
operating system (OS). These
mechanisms include low-level
address space management,
thread management, and
inter-process communication
(IPC).
 Multicomputer Operating
Systems (1)

1.14

General structure of a multicomputer operating system

 Multicomputer Operating
Systems (2)

Alternatives for blocking and buffering in message passing.

 Multicomputer Operating
Systems (3)

Reliable comm.
Synchronization point Send buffer
guaranteed?

Block sender until buffer not full Yes Not necessary

Block sender until message sent No Not necessary

Block sender until message received No Necessary

Block sender until message delivered No Necessary

• Relation between blocking, buffering, and reliable

communications.
 Distributed Shared Memory Systems (1)

a) Pages of address
space distributed
among four
machines

b) Situation after
CPU 1 references
page 10

c) Situation if page
10 is read only
and replication is
used
 Distributed Shared Memory Systems (2)

• False sharing of a page between two independent processes.

1.18
 Network Operating System (1)
• General structure of a network operating system.

1-19
 Network Operating System (2)
• Two clients and a server in a network operating system.

1-20
 Network Operating System (3)
• Different clients may mount the servers in different places.

1.21
Middleware
 Middleware
• Neither DOS or NOS qualifies as Distributed
System.
• Solution:
– Additional layer - middleware
 Positioning Middleware
• General structure of a distributed system as middleware.
 Middleware and Openness
• In an open middleware-based distributed system, the
protocols used by each middleware layer should be
the same, as well as the interfaces they offer to
applications.
Comparison between Systems
Distributed OS
Middleware-
Item Network OS
based OS
Multiproc. Multicomp.

Degree of transparency Very High High Low High

Same OS on all nodes Yes Yes No No

Number of copies of OS 1 N N N

Shared
Basis for communication Messages Files Model specific
memory
Global, Global,
Resource management Per node Per node
central distributed

Scalability No Moderately Yes Varies

Openness Closed Closed Open Open

A comparison between multiprocessor operating systems, multicomputer

operating systems, network operating systems, and middleware based
distributed systems.
 System layers

Applications, services

Middleware

OS: kernel, OS1 OS2

libraries & Processes, threads, Processes, threads,
servers communication, ... communication, ...
Platform

Computer & Computer &

network hardware network hardware

Node 1 Node 2
Middleware and the Operating
System
• Middleware implements abstractions that support
network-wide programming. Examples:
• RPC and RMI (Sun RPC, Corba, Java RMI)
• event distribution and filtering (Corba Event Notification, Elvin)
• resource discovery for mobile and ubiquitous computing
• support for multimedia streaming

• Traditional OS's (e.g. early Unix, Windows 3.0)

– simplify, protect and optimize the use of local resources

• Network OS's (e.g. Mach, modern UNIX, Windows NT)

– do the same but they also support a wide range of
communication standards and enable remote processes
to access (some) local resources (e.g. files).
 What is a distributed OS?
• Presents users (and applications) with an
integrated computing platform that hides the
individual computers.
• Has control over all of the nodes (computers) in
the network and allocates their resources to tasks
without user involvement.
• In a distributed OS, the user doesn't know (or care)
where his programs are running.
• Examples:
• Cluster computer systems
• V system, Sprite, Globe OS
• WebOS (?)
 The support required by middleware and
distributed applications
• OS manages the basic resources of computer systems:
– processing, memory, persistent storage and
communication.
• It is the task of an operating system to:
– raise the programming interface for these resources to a
more useful level:
• By providing abstractions of the basic resources such as:
processes, unlimited virtual memory, files, communication
channels
• Protection of the resources used by applications
• Concurrent processing to enable applications to complete their
work with minimum interference from other applications
– provide the resources needed for (distributed) services
and applications to complete their task:
• Communication - network access provided
• Processing - processors scheduled at the relevant computers
 Core OS functionality
Handles the creation
of the operation
upon processes

Process manager

Communication
Communication between threads
Thread manager attached to different
creation, processes on the
synchronizat same computer
ion and
scheduling. Thread manager Memory manager Management
of physical
Supervisor and virtual
memory
Dispatching of interrupts, system call traps and
other exceptions; control of memory management
unit and hardware
27 caches; processor and floating
point unit register manipulations
 Process address space
N
2
• Regions can be shared
Auxiliary – kernel code
regions – libraries
– shared data & communication
– copy-on-write
• Files can be mapped
– Mach, some versions of UNIX
Stack
• UNIX fork() is expensive
Heap – must copy process's address
space

Text
0
 Creation of a new process
Copy-on-write – a convenient optimization

Process A’s address space Process B’s address space

RB copied
from RA
RA RB

Kernel

Shared
frame
A's page B's page
table table

a) Before
29 write b) After write
*
 Threads concept and
implementation
Process
Thread activations

Activation stacks
(parameters, local variables)

Heap (dynamic storage, 'text' (program code)

objects, global variables)
system-provided resources
(sockets, windows, open files)
 Client and server with threads

Thread 2 makes Input-output

Receipt &
requests to server
queuing

Thread 1
generates
results
Requests
N threads

Client Server

The 'worker pool' architecture

 Alternative server threading
architectures
server server server
process workers process per-connection threads process per-object threads

I/O I/O remote

remote remote
objects
objects objects

a. Thread-per-request b. Thread-per-connection c. Thread-per-object

Implemented by the server-side ORB in CORBA

(a) would be useful for UDP-based service
(b) is the most commonly used - matches the TCP connection model
(c) is used where the service is encapsulated as an object. E.g. could have
multiple shared whiteboards with one thread each. Each object has only
one thread, avoiding the need for thread synchronization within objects.
 Java thread constructor and
management methods
Methods of objects that inherit from class Thread
• Thread(ThreadGroup group, Runnable target, String name)
• Creates a new thread in the SUSPENDED state, which will belong to
group and be identified as name; the thread will execute the run()
method of target.
• setPriority(int newPriority), getPriority()
• Set and return the thread’s priority.
• run()
• A thread executes the run() method of its target object, if it has one,
and otherwise its own run() method (Thread implements Runnable).
• start()
• Change the state of the thread from SUSPENDED to RUNNABLE.
• sleep(int millisecs)
• Cause the thread to enter the SUSPENDED state for the specified time.
• yield()
• Enter the READY state and invoke the scheduler.
• destroy()
• Destroy the thread.
34
*
 Java thread synchronization
calls
• thread.join(int millisecs)
• Blocks the calling thread for up to the specified time until thread has terminated.
• thread.interrupt()
• Interrupts thread: causes it to return from a blocking method call such as sleep().
• object.wait(long millisecs, int nanosecs)
• Blocks the calling thread until a call made to notify() or notifyAll() on object wakes the
thread, or the thread is interrupted, or the specified time has elapsed.
• object.notify(), object.notifyAll()
• Wakes, respectively, one or all of any threads that have called wait() on object.

object.wait() and object.notify() are very similar to the semaphore operations. E.g. a worker
thread in Figure 6.5 would use queue.wait() to wait for incoming requests.
synchronized methods (and code blocks) implement the monitor abstraction. The operations
within a synchronized method are performed atomically with respect to other synchronized
methods of the same object. synchronized should be used for any methods that update the
state of an object in a threaded35environment.
*
Java thread synchronization
calls

Input-output
Receipt &
queuing

Requests
N threads

Server

36
 Threads implementation
• Threads can be implemented:
– in the OS kernel (Win NT, Solaris, Mach)
– at user level (e.g. by a thread library: C threads, pthreads),
or in the language (Ada, Java).
• lightweight - no system calls
• modifiable scheduler
• low cost enables more threads to be employed
• not pre-emptive
• can exploit multiple processors
– page fault blocks all threads
• Java can be implemented either way
– hybrid approaches can gain some advantages of both
• user-level hints to kernel scheduler
• heirarchic threads (Solaris)
• event-based (SPIN, FastThreads)
Support for communication and
invocation
• The performance of RPC and RMI mechanisms is critical
for effective distributed systems.
– Typical times for 'null procedure call':
– Local procedure call < 1 microseconds
10,000 times slower!
– Remote procedure call ~ 10 milliseconds
– 'network time' (involving about 100 bytes transferred, at 100
megabits/sec.) accounts for only .01 millisecond; the remaining delays
must be in OS and middleware - latency, not communication time.
• Factors affecting RPC/RMI performance
– marshalling/unmarshalling + operation despatch at the server
– data copying:- application -> kernel space -> communication buffers
– thread scheduling and context switching:- including kernel entry
– protocol processing:- for each protocol layer
– network access delays:- connection setup, network latency
 Implementation of invocation
mechanisms
• Most invocation middleware (Corba, Java RMI, HTTP) is
implemented over TCP
– For universal availability, unlimited message size and reliable transfer
– Sun RPC (used in NFS) is implemented over both UDP and TCP and
generally works faster over UDP
• Research-based systems have implemented much more
efficient invocation protocols, E.g.
– Firefly RPC (see www.cdk3.net/oss)
– Amoeba's doOperation, getRequest, sendReply primitives
(www.cdk3.net/oss)
– LRPC [Bershad et. al. 1990], described on pp. 237-9)..
• Concurrent and asynchronous invocations
– middleware or application doesn't block waiting for reply to each
invocation
Invocations between address
spaces
(a) System call Thread Control transfer via
trap instruction

Control transfer via

privileged instructions
User Kernel
Protection domain
(b) RPC/RMI (within one computer) boundary

Thread 1 Thread 2

User 1 Kernel User 2

(c) RPC/RMI (between computers)

Thread 1 Network Thread 2

User 1 User 2
Kernel 1 Kernel 2
 Bershad's LRPC
• Uses shared memory for interprocess communication
– while maintaining protection of the two processes
– arguments copied only once (versus four times for convenitional RPC)
• Client threads can execute server code
– via protected entry points only (uses capabilities)
• Up to 3 x faster for local invocations

42
A lightweight remote procedure
call
Client Server

A stack
A

1. Copy args 4. Execute procedure and copy results

User stub stub

Kernel
2. Trap to Kernel 3. Upcall 5. Return (trap)
Operating System architecture
Monolithic kernel and microkernel

S4 .......

S1 S2 S3 S4 .......
.......
S1 S2 S3 Kernel Kernel

Monolithic Kernel Microkernel

Middleware
Language Language OS emulation
support support subsystem
subsystem subsystem ....
Microkernel
Hardware

The microkernel supports middleware via subsystems

 Advantages and
disadvantages of microkernel
• flexibility and extensibility
– services can be added, modified and
debugged
– small kernel -> fewer bugs
– protection of services and resources is still
maintained
• service invocation expensive
– unless LRPC is used
– extra system calls by services for access to
protected resources
 Summary
• The OS provides local support for the implementation of
distributed applications and middleware:
– Manages and protects system resources (memory, processing,
communication)
– Provides relevant local abstractions:
• files, processes
• threads, communication ports
• Middleware provides general-purpose distributed
abstractions
– RPC, DSM, event notification, streaming
• Invocation performance is important
– it can be optimized, E.g. Firefly RPC, LRPC
• Microkernel architecture for flexibility
– The KISS principle ('Keep it simple – stupid!')
• has resulted in migration of many functions out of the OS