Definition of a Distributed System

• A distributed system is:

A collection of independent
computers that appears to its
users as a single coherent
system.
 Distributed System Organization

1.1

• Example of middleware-based organization of a distributed system.

• The thickness of the middleware layer can range from extremely thin to
very thick depending on the degree of integration of a particular system
Hardware Organizations

1.6

Now this model dominates, but

it may change in the future.
 Hardware Organization
• Most popular model today:
– Each machine has its own memory, storage, etc
– Machines are connected by a network and communicate
with network packets.
• Shared-memory model:
– Each machine can access all memory with load/store
instructions: building a distributed program is not much
different from building a multi-threaded program
– One can emulate load/store instructions with network
packets, converting the above model into this model
– Still an open research topic.
Terminologies in distributed systems
• Throughput: how many requests can the system
process per second?
• Latency: how long does the system take to process
one request?
• Scalability: can the system gain more throughput
by adding more machines?

• Discussion: their relationship?

 Analogy with a highway
• Throughput: determined by the number of lanes,
car speed, and the distance between cars
• Latency: determined by the distance between
source and destination and car speed (assuming no
jam)
• Traffic jam (demand > throughput) => Queuing
delay => Longer latency
• Scalability: can we get more throughput by adding
lanes?
Terminologies in distributed systems
• Consistency: what is the correct behavior of the
system?

• Availability: how quickly can the system respond

to clients’ requests? Usually measured by tail
latency.
• Liveness: can the system eventually finish
processing a request?
• Durability: can the system lose data?
Terminologies in distributed systems
• Availability, Liveness, and Durability
– Liveness is a YES/NO property. If the system takes one
year to process a client’s request, it is still live.
– Availability is a quantitative property. A system taking
one year to process a request has bad availability.
– A system may be durable but not live: the system may
have your data but you cannot access it for whatever
reason. But if a system is not durable, then it is usually
not live (in theory, you can build a live but not durable
system by giving the user garbage data, but of course
this is not desirable).
Terminologies in distributed systems
• Fault tolerance:
– Consistency, liveness, and durability are all trivial if not
considering failures.
– In a large distributed systems, failures are common:
machines can crash; network links may be lost; disks
can have permanent failures; ……
– How to achieve consistency, liveness, and durability
despite these failures becomes challenging
 Process Communication

• Process communication today tends to be specialized

along application domains
– Universal communication support: TCP/IP protocol, stack
– High performance computing: MPI
– Distributed programming: RPC, RMI, Object brokers, web
services
– Digital Multimedia: streaming protocols
• Conceptually they can be thought of as a specialization
of the early process communication concepts
 Process Communication

• Message Passing Model

– Socket, MPI, …
• Remote Procedure Call (RPC) model
– Java RMI, Google protobuf, Apache Thrift, …
• Remote DMA (RDMA) model
• In today’s switch-based networks, all models pass
messages internally, but provide different interfaces to
users.
 Message Passing model

• Distributed systems -> processes interact via messages

– Distributed applications on the Web
– Parallel programs consisting of cooperating processes
• Communication is explicit
– Programmer uses send and receive operations, involved in
details of data transport and data synchronization
• Messages have two functions
– exchange data
– synchronize their executions
 Evolution of Layered Protocols
• Many new protocols developed as part of new applications – FTP, HTTP, SSH, etc,
not originally foreseen by the OSI architects
• New middleware layer replaces obsolete Presentation and Session layers
 Berkeley Sockets (1)
• Socket primitives for TCP/IP.

Primitive Meaning

Socket Create a new communication endpoint

Bind Attach a local address to a socket

Listen Announce willingness to accept connections

Accept Block caller until a connection request arrives

Connect Actively attempt to establish a connection

Send Send some data over the connection

Receive Receive some data over the connection

Close Release the connection

 Berkeley Sockets (2)

• Connection-oriented communication pattern using sockets.

 Blocking vs Non-blocking
• Blocking mode:
– A write returns when the sender receives ACK from receiver
– A read returns if it receives at least one byte; otherwise block

• Non-blocking mode: use select/epoll

– Write may fail immediately if there is no buffer
– Read will return NULL immediately if there is no data
– Use select/epoll to check whether it is OK to send or read
 The Remote Procedure Call Model

• Distributed systems -> processes interact via messages

• However the communication is implicit

– RPC looks and smells like a local procedure call
– Conceptually simple to implement, the devil is in the details
• A RPC has multiple functions
– Data transport
– Synchronization of executions of processes
 Conventional Procedure Call

a) Parameter passing in a local procedure call: the stack before the call to
read
b) The stack while the called procedure is active
 Client and Server Stubs

• Principle of RPC between a client and server program.

• Client and server stubs (intermediate auxiliary functions) used to build
the illusion of a local procedure call
• Main task of stubs: packing and unpacking of parameters and results in
messages
Steps of a Remote Procedure Call

1. Client procedure calls client stub in a normal

way
2. Client stub builds message, calls local OS
3. Client's OS sends message to remote OS
4. Remote OS gives message to server stub
5. Server stub unpacks parameters, calls server
6. Server does work, returns result to the stub
7. Server stub packs it in message, calls local OS
8. Server's OS sends message to client's OS
9. Client's OS gives message to client stub
10. Stub unpacks result, returns to client
 Passing Value Parameters (1)
• Steps involved in doing remote computation through RPC
 Passing Value Parameters (2)
A major complication is the different byte ordering adopted in different processor
architectures
(a) Original message on the Pentium (little endian)
(b) The message after receipt on the SPARC (big endian)
(c) The message after being inverted.
The little numbers in boxes indicate the address of each byte
Parameter Specification and Stub Generation

a) A procedure
b) The corresponding message.
 Asynchronous RPC (1)

2-12

a) The interconnection between client and server in a traditional RPC

b) The interaction using asynchronous RPC
 Asynchronous RPC (2)
• A client and server interacting through two asynchronous RPCs

2-13
 RPC libraries
• There are libraries that can automatically generate
stubs to pack/unpack messages

• Examples:
– Java JMI
– Apache Thrift
– Google protobuf
– …
 Topics of Distributed System
Section
• Distributed deadlock detection
– Global state and distributed snapshot
– Logical clock and vector clock, …
• Distributed mutual exclusion
• Distributed transaction execution
• Fault tolerance
– Quorum-based approach
– Leader-based approach: Paxos and PBFT
• Consistency