Parallel and Distributed

Computing
FALL -2022
Dr. Zeshan Iqbal
Lecture 1: Welcome, and Introduction

What This Course is Really About

• Distributed Systems
• How to Design Algorithms for them
• How to Design The Systems
• How they work in real life
• How to build real distributed systems

2
 Our Main Goal Today

To Define the Term Distributed System

Can you name some examples of

Operating Systems?

4
 Can you name some examples of
Operating Systems?
…
Linux WinXP Vista 7/8 Unix FreeBSD Mac OSX
2K Aegis Scout Hydra Mach SPIN
OS/2 Express Flux Hope Spring
AntaresOS EOS LOS SQOS LittleOS TINOS
PalmOS WinCE TinyOS iOS
…

What is an Operating System?

6
 What is an Operating System?

• User interface to hardware (device driver)

• Provides abstractions (processes, file system)
• Resource manager (scheduler)
• Means of communication (networking)
• …

FOLDOC definition
(FOLDOC = Free On-Line Dictionary of Computing)
Operating System - The low-level software
which handles the interface to peripheral
hardware, schedules tasks, allocates storage,
and presents a default interface to the user
when no application program is running.

8
 Can you name some examples of
Distributed Systems?

Can you name some examples of

Distributed Systems?

• Client-Server (NFS)
• The Web
• The Internet
• A wireless network
• DNS
• Gnutella or BitTorrent (peer to peer overlays)
• A “cloud”, e.g., Amazon EC2/S3, Microsoft Azure
• A datacenter, e.g., NCSA, a Google datacenter, AWS

10

10
 What is a Distributed System?

11

11

FOLDOC definition

A collection of (probably heterogeneous) automata whose

distribution is transparent to the user so that the system
appears as one local machine. This is in contrast to a
network, where the user is aware that there are several
machines, and their location, storage replication, load
balancing and functionality is not transparent. Distributed
systems usually use some kind of client-server
organization.

12

12
 Textbook definitions
• A distributed system is a collection of independent
computers that appear to the users of the system as a
single computer.
[Andrew Tanenbaum]

• A distributed system is several computers doing

something together. Thus, a distributed system has three
primary characteristics: multiple computers,
interconnections, and shared state.
[Michael Schroeder]
13

13

Unsatisfactory
• Why are these definitions short?
• Why do these definitions look inadequate to us?
• Because we are interested in the insides of a
distributed system
– design and implementation
– Maintenance
– Algorithmics (“protocols” or “distributed algorithms”)

14

14
 “I shall not today attempt further to define the kinds of material I
understand to be embraced within that shorthand description; and
perhaps I could never succeed in intelligibly doing so. But I know it
when I see it, and the motion picture involved in this case is not that.”

[Potter Stewart, Associate Justice, US Supreme Court (talking about

his interpretation of a technical term laid down in the law, case
Jacobellis versus Ohio 1964) ]

15

15

Which is a Distributed System – (A) or (B)?

(A)

(A) Facebook Social Network Graph among humans

Source: https://www.facebook.com/note.php?note_id=469716398919 16

16
 (B)

(B) The Internet (Internet Mapping Project, color coded by ISPs)

17

17

A working definition for us

A distributed system is a collection of entities, each of
which is autonomous, programmable, asynchronous
and failure-prone, and which communicate through an
unreliable communication medium.

• Entity=a process on a device (PC, PDA)

• Communication Medium=Wired or wireless network
• Our interest in distributed systems involves
– design and implementation, maintenance, algorithmics

18

18
 Gnutella Peer to Peer System

What are the “entities”

(nodes)?

Source: GnuMap Project What is the

communication medium
(links)?

19

19

Web Domains
What are the
“entities”
(nodes)?

What is the
communication
medium
(links)?

Source: http://www.vlib.us/web/worldwideweb3d.html 20

20
 Datacenter

What are the “entities”

(nodes)?

What is the
communication medium
(links)?
21

21

The Internet – Quick Refresher

• Underlies many distributed systems.
• A vast interconnected collection of computer networks of many types.
• Intranets – subnetworks operated by companies and organizations.
• Intranets contain LANs (local area networks).
• WAN – wide area networks, consists of subnets (intranets, LANs,
etc.)
• ISPs – companies that provide modem links and other types of
connections to users.
• Intranets (actually the ISPs’ core routers) are linked by backbones –
network links of large bandwidth, such as satellite connections, fiber
optic cables, and other high-bandwidth circuits.
• UC2B? Google Fiber? (MAN = Metropolitan Area Networks)
22

22
 An Intranet & a distributed system
email server De sktop
computers
print and other serv ers

Local area
Running over this Web server network
Intranet is a distributed
file system
email server
print
File server
other s ervers

the res t of
the Internet
router/firewall
prevents unauthorized messages from leaving/entering;
implemented by filtering incoming and outgoing messages
via firewall “rules” (configurable)

23

23

Does our Working Definition work for the http

Web?

A distributed system is a collection of entities, each of

which is autonomous, programmable, asynchronous
and failure-prone, and that communicate through an
unreliable communication medium.

• Entity=a process on a device (PC, PDA)

• Communication Medium=Wired or wireless network
• Our interest in distributed systems involves
– design and implementation, maintenance, study, algorithmics
24

24
 “Important” Distributed Systems
Issues
• No global clock; no single global notion of the correct time
(asynchrony)
• Unpredictable failures of components: lack of response may be due
to either failure of a network component, network path being
down, or a computer crash (failure-prone, unreliable)
• Highly variable bandwidth: from 16Kbps (slow modems or Google
Balloon) to Gbps (Internet2) to Tbps (in between DCs of same big
company)
• Possibly large and variable latency: few ms to several seconds
• Large numbers of hosts: 2 to several million

25

25

Many Interesting Design Problems

• Real distributed systems
– Cloud Computing, Peer to peer systems, Hadoop, key-value
stores/NoSQL, distributed file systems, sensor networks,
measurements, graph processing, stream processing, …
• Classical Problems
– Failure detection, Asynchrony, Snapshots, Multicast,
Consensus, Mutual Exclusion, Election, …
• Concurrency
– RPCs, Concurrency Control, Replication Control, Paxos, …
• Security
– ACLs, Capabilities, …
• Others…
•
26

26
 Typical Distributed Systems Design
• Common Goals:
Goals
– Heterogeneity – can the system handle a large variety of types of PCs
and devices?
– Robustness – is the system resilient to host crashes and failures, and
to the network dropping messages?
– Availability – are data+services always there for clients?
– Transparency – can the system hide its internal workings from the
users? (warning: term means the opposite of what the name implies!)
– Concurrency – can the server handle multiple clients simultaneously?
– Efficiency – is the service fast enough? Does it utilize 100% of all
resources?
– Scalability – can it handle 100 million nodes without degrading
service? (nodes=clients and/or servers) How about 6 B? More?
– Security – can the system withstand hacker attacks?
– Openness – is the system extensible? 27

27

“Important” Issues
• If you’re already complaining that the list of topics
we’ve discussed so far has been perplexing…
– You’re right!
– It was meant to be (perplexing)

• The Goal for the Rest of the Course: see enough

examples and learn enough concepts so these topics
and issues will make sense
– We will revisit many of these slides in the very last lecture of
the course!

28

28
 “Concepts”?
• Which of the following inventions do you
think is the most important?
1. Car
2. Wheel
3. Bicycle

“What lies beneath?” Concepts!

29

29

How will We Learn?

All this information is contained in handout on LMS website:
“Cloud Computing- Resources”

• Web:
• Textbook, Recommended but not Required
– Colouris, Dollimore, Kindberg and Blair (5th edition)
– If you use a different or older version, be sure to check
problem numbers, etc.
– Textbook is a great source of practice problems for the exams
(midterm and final)

• Lectures
30

30