Distributed Systems

Processes
Chapter 3

1
 Course/Slides Credits
Note: all course presentations are based on those
developed by Andrew S. Tanenbaum and
Maarten van Steen. They accompany their
"Distributed Systems: Principles and
Paradigms" textbook (1st & 2nd editions).
http://www.prenhall.com/divisions/esm/app/aut
hor_tanenbaum/custom/dist_sys_1e/index.html
And additions made by Paul Barry in course
CW046-4: Distributed Systems
http://glasnost.itcarlow.ie/~barryp/net4.html
2
 Processes
• Communication takes place between processes.
• But, what’s a process? “A program in execution”.
• Traditional operating systems: concerned with the
“local” management and scheduling of processes.
• Modern distributed systems: a number of other
issues are of equal importance.
• There are three main areas of study:
1. Threads and virtualization within clients/servers
2. Process and code migration
3. Software agents
3
 Introduction to Threads
• Modern OSs provide “virtual processors”
within which programs execute.
• A programs execution environment is
documented in the process table and assigned
a PID.
• To achieve acceptable performance in
distributed systems, relying on the OS’s idea
of a process is often not enough –
finer granularity is required.
4 • The solution: Threading.
 Problems with Processes
• Creating and managing processes is generally
regarded as an expensive task (fork system call).
• Making sure all the processes peacefully co-exist on
the system is not easy (as concurrency transparency
comes at a price).
• Threads can be thought of as an “execution of a part
of a program (in user-space)”.
• Rather than make the OS responsible for concurrency
transparency, it is left to the individual application to
manage the creation and scheduling of each thread.
5
 Important Implications
• Two Important Implications:
1. Threaded applications often run faster than non-
threaded applications (as context-switches
between kernel and user-space are avoided).
2. Threaded applications are harder to develop
(although simple, clean designs can help here).
• Additionally, the assumption is that the
development environment provides a
Threads Library for developers to use
(most modern environments do).
6
 Thread Usage in Non-distributed Systems

Context switching as the result of IPC

7
 Thread Implementation

Combining kernel-level lightweight processes

8 and user-level threads
 Threads in Non-Distributed Systems

• Advantages:
1. Blocking can be avoided
2. Excellent support for multi-processor
systems (each running their own thread).
3. Expensive context-switches can be
avoided.
4. For certain classes of application, the
design and implementation is made
considerably easier.
9
 Threads in Distributed Systems
• Important characteristic: a blocking call in a
thread does not result in the entire process
being blocked.
• This leads to the key characteristic of threads
within distributed systems:
– “We can now express communications in the form
of maintaining multiple logical connections at the
same time (as opposed to a single, sequential,
blocking process).”

10
 Example: MT Clients and Servers
• Mutli-Threaded Client: to achieve acceptable
levels of perceived performance, it is often
necessary to hide communications latencies.
• Consequently, a requirement exists to start
communications while doing something else.
• Example: modern Web browsers.
• This leads to the notion of “truly parallel
streams of data” arriving at a multi-threaded
client application.
11
 Example: MT-Servers
• Although threading is useful on clients, it is
much more useful in distributed systems
servers.
• The main idea is to exploit parallelism to attain
high performance.
• A typical design is to organize the server as a
single “dispatcher” with multiple threaded
“workers”, as diagrammed overleaf.

12
 Multithreaded Servers (1)

A multithreaded server organized in a

13 dispatcher/worker model
 Multithreaded Servers (2)

Three ways to construct a server

14
 The Role of Virtualization in
Distributed Systems

(a) General organization between a program, interface, and system

(b) General organization of virtualizing system A on top of system
B
15
 Architectures of Virtual Machines (1)
• There are interfaces at different levels.
• An interface between the hardware and
software, consisting of machine instructions
– that can be invoked by any program.
• An interface between the hardware and
software, consisting of machine instructions
– that can be invoked only by privileged
programs, such as an operating system.

16
 Architectures of Virtual Machines (2)
• An interface consisting of system calls
as offered by an operating system.
• An interface consisting of library calls
– generally forming what is known as an
Application Programming Interface
(API).
– In many cases, the aforementioned
system calls are hidden by an API.
17
 Architectures of Virtual Machines (3)

Various interfaces offered by computer systems

18
 Architectures of Virtual Machines (4)

(a) A process virtual machine, with multiple

instances of (application, runtime) combinations
19
 The Java Virtual Machine

20
 Architectures of Virtual Machines (5)

(b) A Virtual Machine Monitor (VMM), with

multiple instances of (applications, operating
21 system) combinations.
 More on Clients
• What’s a client?
• Definition: “A program which interacts with a
human user and a remote server.”
• Typically, the user interacts with the client
via a GUI.
• Of course, there’s more to clients than simply
providing a UI. Remember the multi-tiered
levels of the Client/Server architecture from
earlier …
24
 Generic Client/Server Environment

25
 Networked User Interfaces (1)

29 (a) A networked application with its own protocol

 Networked User Interfaces (2)

(b) A general solution to allow access

30 to remote applications
 Example: The X Window System

31
The basic organization of the X Window System
 Client-Side Software for
Distribution Transparency

Transparent replication of a server

32 using a client-side solution
 More on Servers
• What’s a server?
• Definition: “A process that implements a
specific service on behalf of a collection of
clients”.
• Typically, servers are organized to do one of
two things:
1. Wait
2. Service
… wait … service … wait … service … wait …
33
 Servers: Iterative and Concurrent
• Iterative: server handles request, then returns
results to the client; any new client requests
must wait for previous request to complete
(also useful to think of this type of server as
sequential).
• Concurrent: server does not handle the request
itself; a separate thread or sub-process handles
the request and returns any results to the client;
the server is then free to immediately service
the next client (i.e., there’s no waiting, as
34
service requests are processed in parallel).
 ”Server “States
• Stateless servers – no information is maintained on
the current “connections” to the server. The Web is
the classic example of a stateless service. As can be
imagined, this type of server is easy to implement.
• Stateful servers – information is maintained on the
current “connections” to the server. Advanced file
servers, where copies of a file can be updated
“locally”, then applied to the main server (as it knows
the state of things) - more difficult to implement.
• But, what happens if something crashes? (more on
this later).
35
 ?”Problem: Identifying “end-points
• How do clients know which end-point
(or port) to contact a server at?
How do they “bind” to a server?
– Statically assigned end-points (IANA).
– Dynamically assigned end-points (DCE).
– A popular variation:
• the “super-server” (inetd on UNIX).

36
 General Design Issues (1)

(a) Client-to-server binding using a daemon

37
 General Design Issues (2)

38 (b) Client-to-server binding using a superserver

 A Special Type: Object Servers
• A server tailored to support distributed objects.
• Does not provide a specific service.
• Provides a facility whereby objects can be
remotely invoked by non-local clients.
• Consequently, object servers are highly
adaptable.
• “A place where objects live”.

39
 Object Adapter

Organization of an object server supporting different

40 activation policies.
 Server Clusters (1)

41 The general organization of a three-tiered server cluster

 Server Clusters (2)

42 The principle of TCP handoff

 Distributed Servers

Route optimization in a distributed server

43
 Managing Server Clusters

44 The basic organization of a PlanetLab node

 PlanetLab (1)
• PlanetLab management issues:
• Nodes belong to different organizations.
– Each organization should be allowed to specify who
is allowed to run applications on their nodes,
– And restrict resource usage appropriately.
• Monitoring tools available assume a very specific
combination of hardware and software.
– All tailored to be used within a single organization.
• Programs from different slices but running on the
same node should not interfere with each other.
45
 PlanetLab (2)

The management relationships

46 between various PlanetLab entities
 PlanetLab (3)
• Relationships between PlanetLab entities:
• A node owner puts its node under the regime
of a management authority, possibly
restricting usage where appropriate.
• A management authority provides the
necessary software to add a node to
PlanetLab.
• A service provider registers itself with a
management authority, trusting it to provide
47 well-behaving nodes.
 PlanetLab (4)
• A service provider contacts a slice authority
to create a slice on a collection of nodes.
• The slice authority needs to authenticate the
service provider.
• A node owner provides a slice creation
service for a slice authority to create slices. It
essentially delegates resource management to
the slice authority.
• A management authority delegates the
creation of slices to a slice authority.
48
 Process and Code Migration

• Under certain circumstances, in addition

to the usual passing of data, passing code
(even while it is executing) can greatly
simplify the design of a DS.
• However, code migration can be
inefficient and very costly.
• So, why migrate code?

49
 Reasons for Migrating Code

• Why? Biggest single reason: better

performance.
• The big idea is to move a compute-
intensive task from a heavily loaded
machine to a lightly loaded machine
“on demand” and “as required”.

50
 Example of Process Migration

51
 Code Migration Examples
• Moving (part of) a client to a server –
processing data close to where the data resides.
It is often too expensive to transport an entire
database to a client for processing, so move the
client to the data.
• Moving (part of) a server to a client –
checking data prior to submitting it to a server.
The use of local error-checking (using
JavaScript) on Web forms is a good example of
this type of processing. Error-check the data
52
close to the user, not at the server.
 Classic” Code Migration Example“

• Searching the Web by “roaming”.

• Rather than search and index the Web by
requesting the transfer of each and every
document to the client for processing, the
client relocates to each site and indexes the
documents it finds “in situ”. The index is then
transported from site to site, in addition to the
executing process.

53
 Another Big Advantage: Flexibility

The principle of dynamically configuring a client to communicate to a

server. The client first fetches the necessary software, and then invokes
54 the server. This is a very flexible approach.
 Major Disadvantage
• Security Concerns.
• “Blindly trusting that the downloaded code
implements only the advertised interface while
accessing your unprotected hard-disk and does
not send the juiciest parts to heaven-knows-
where may not always be such a good idea”.

55
 Code Migration Models

• A running process consists of three

“segments”:
1. Code – instructions.
2. Resource – external references.
3. Execution – current state.

56
 Migration in Heterogeneous Systems
• Three ways to handle migration (which can be
combined):
1. Pushing memory pages to the new machine and
resending the ones that are later modified during the
migration process.
2. Stopping the current virtual machine; migrate
memory, and start the new virtual machine.
3. Letting the new virtual machine pull in new pages as
needed, that is, let processes start on the new virtual
machine immediately and copy memory pages on
57 demand.
 Code Migration Characteristics

• Weak Mobility: just the code is moved –

and it always restarts from its initial state.
– e.g., Java Applets.
– Comment: simple implementation, but limited
applicability.
• Strong Mobility: code & state is moved –
and execution restarts from the next statement.
– e.g., D’Agents.
– Comment: very powerful, but hard to implement.
58
 More Characteristics
• Sender-Initiated vs. Receiver-Initiated.
• Which side of the communication starts
the migration?
1. The machine currently executing the code
(known as sender-initiated)
2. The machine that will ultimately execute
the code (known as receiver-initiated).

59
 ?How Does the Migrated Code Run
• Another issue surrounds where the migrated
code executes:
1. Within an existing process (possibly as a thread)
2. Within it’s own (new) process space.
• Finally, strong mobility also supports the
notion of “remote cloning”: an exact copy of
the original process, but now running on a
different machine.

60
 Models for Code Migration

61 Alternatives for code migration

 ?What About Resources
• This is tricky.
• What makes code migration difficult is the
requirement to migrate resources.
• Resources are the external references that
a process is currently using, and includes
(but is not limited to):
– Variables, open files, network connections,
printers, databases, etc...

62
 Types of Process-to-Resource Binding
• Strongest: Binding-by-Identifier (BI) –
precisely the referenced resource, and nothing
else, has to be migrated.
• Binding-by-Value (BV) –
weaker than BI, but only the value of the
resource need be migrated.
• Weakest: Binding-by-Type (BT) –
nothing is migrated, but a resource of a specific
type needs to be available after migration
(e.g., a printer).
63
 More Resource Classification
• Resources are further distinguished as one of:
1. Unattached: a resource that can be moved easily
from machine to machine.
2. Fastened: migration is possible, but at a high cost.
3. Fixed: a resource is bound to a specific machine
or environment, and cannot be migrated.
• Refer to following diagram for a good
summary of resource-to-binding
characteristics (to find out what to do with
which resource when).
64
 Migration and Local Resources

Actions to be taken with respect to the references

to local resources when migrating code to
65 another machine
 Migration in Heterogeneous DS’s

3-15

Using a migration stack: the principle of maintaining a migration stack to

support migration of an execution segment in a heterogeneous environment.
66 Usually requires changes to the programming language and its environment.
 Overview of Code Migration in D'Agents (1)
proc factorial n {
if ($n  1) { return 1; } # fac(1) = 1
expr $n * [ factorial [expr $n – 1] ] # fac(n) = n * fac(n – 1)
}
set number … # tells which factorial to compute
set machine … # identify the target machine

agent_submit $machine –procs factorial –vars number –script {factorial $number }

agent_receive … # receive the results (left unspecified for simplicity)

A simple example of a Tel agent in D'Agents submitting a

script to a remote machine (adapted from [Gray 95])
67
 Overview of Code Migration in D'Agents (2)
all_users $machines
proc all_users machines {
set list "" # Create an initially empty list
foreach m $machines { # Consider all hosts in the set of given machines
agent_jump $m # Jump to each host
set users [exec who] # Execute the who command
append list $users # Append the results to the list
}
return $list # Return the complete list when done
}
set machines … # Initialize the set of machines to jump to
set this_machine # Set to the host that starts the agent
# Create a migrating agent by submitting the script to this machine, from where
# it will jump to all the others in $machines.
agent_submit $this_machine –procs all_users
-vars machines
-script { all_users $machines }
agent_receive … #receive the results (left unspecified for simplicity)

An example of a Tel agent in D'Agents migrating to different machines

where it executes the UNIX who command.
68
 Implementation Issues (1)

69 The architecture of the D'Agents system.

 Implementation Issues (2)

Description Status

Variables needed by the interpreter of an agent Global interpreter variables

Return codes, error codes, error strings, etc. Global system variables

User-defined global variables in a program Global program variables

Definitions of scripts to be executed by an agent Procedure definitions

Stack of commands currently being executed Stack of commands

Stack of activation records, one for each running command Stack of call frames

70 The parts comprising the state of an agent in D'Agents.

 Software Agents
• What is a software agent?
– “An autonomous unit capable of performing a task
in collaboration with other, possibly remote,
agents”.
• The field of Software Agents is still immature,
and much disagreement exists as to how to
define what we mean by them.
• However, a number of types can be identified.

71
 Types of Software Agents
1. Collaborative Agent – also known as “multi-agent
systems”, which can work together to achieve a
common goal (e.g., planning a meeting).
2. Mobile Agent – code that can relocate and continue
executing on a remote machine.
3. Interface Agent – software with “learning abilities”
(that damned MS paperclip, and the ill-fated “bob”).
4. Information Agent – agents that are designed to
collect and process geographically dispersed data
and information.
72
 Software Agents in Distributed Systems
Common to
Description Property
all agents?

Can act on its own Yes Autonomous

Responds timely to changes in its environment Yes Reactive

Initiates actions that affects its environment Yes Proactive

Can exchange information with users and other agents Yes Communicative

Has a relatively long lifespan No Continuous

Can migrate from one site to another No Mobile

Capable of learning No Adaptive

Some important properties by which different types of

73 agents can be distinguished.
 Agent Technology - Standards
• The general model of an agent platform has
been standardized by FIPA (“Foundation
for Intelligent Physical Agents”) located at
http://www.fipa.org
• Specifications include:
– Agent Management Component.
– Agent Directory Service.
– Agent Communication Channel.
– Agent Communication Language.
74
 Agent Technology

The general model of an agent platform

75 (adapted from [FIPA 1998])
 Agent Communication Languages (1)

Message Content Description Message purpose

Proposition Inform that a given proposition is true INFORM

Proposition Query whether a given proposition is true QUERY-IF

Expression Query for a give object QUERY-REF

Proposal specifics Ask for a proposal CFP

Proposal Provide a proposal PROPOSE
Proposal ID Tell that a given proposal is accepted ACCEPT-PROPOSAL
Proposal ID Tell that a given proposal is rejected REJECT-PROPOSAL
Action specification Request that an action be performed REQUEST
Reference to source Subscribe to an information source SUBSCRIBE

Examples of different message types in the FIPA ACL [ FIPA 1998], giving the
purpose of a message, along with the description of the actual message content.
76
 Agent Communication Languages (2)

Value Field

INFORM Purpose

max@http://fanclub-beatrix.royalty-spotters.nl:7239 Sender

elke@iiop://royalty-watcher.uk:5623 Receiver

Prolog Language

genealogy Ontology

female(beatrix),parent(beatrix,juliana,bernhard) Content

A simple example of a FIPA ACL message sent between two agents using
77 Prolog to express genealogy information.