Parallel & Distributed Computing

Dr. Muhammad Khuram Shahzad

Office: A-308
Class: BSCS-11C-2k21
SEECS Faculty Block
Parallel & Distributed Computing - Spring 2024

Introduction: 1-2
 3

About Instructor
Dr. M Khuram Shahzad
Associate Professor - 01.2019 - 02.2024 – NUST University, Pakistan.
Assistant Professor – 09.2017 - 05.2018 – Keinyung University, South Korea.
PhD (AI) – SungKyunKwan Uni. (QS #100), S.K, 2016.
Postdoc – SungKyunKwan Uni. , SK, Dec. 2017.
Head PG AI&DS Research & UG AI&DS Research Groups

Collaborations
▪ Sejong University, S.Korea
Publications
▪ DuDu Information Technologies, Inc., S.Korea
• IF Journals: 18 ▪ University of Alabama at Birmingham, US
• N-IF Journals: 07 ▪ King Saud University, KSA
• Conference : 10 ▪ Inha University, S.Korea
• Datasets : 02
Student Supervision
Research Areas Problem Solvers PhD: 1, MS: 11 (2 ongoing) UG: 8 (1
on going)
• Wireless Networks • Artificial
• Health Care Intelligence
Teaching (NUST and S.Korea)
• Network Simmulatiom • Data Science
Sustainable • Graph Theory UG: 15 courses
•
Development Optimization PG: 5 courses
•
Contact: Office A-308 (4-5PM, VOIP 2557, email: mkhuram.shahzad@seecs.edu.pk
Course outline [½]

4
Course outline [2/2]

5
 Text & Reference Books
● Textbook
Distributed Systems, Principles and Paradigms by Tanenbaum, Van Steen, Second Edition, ISBN-978-81-203-
3498
● Reference Books:
Distributed Systems: Concepts and Design by George Coulouris, Jean Dollimore, and Tim Kindberg, Addison
Wesley, 5th edition, 2012.
Distributed Systems, An algorithmic approach, Sukumar Ghosh, Chapman & Hall/CRC Computer and
Information Science Series, ISBN 10:1-58488-564-5
Parallel and distributed simulation systems, Richard Fujimoto, ISBN 0-471-18383-0
Any relevant book on Solr, AWS, Hadoop, and Cloud Computing
Research papers if required.

6
Assessments

7
 Performance Needed For Big Problems
❏ The FLOP. FLoating point Operation - # flops/ sec is the standard metric for
computing power
❏ Example: Global Climate Modeling
❏ Divide the world into a grid
❏ Solve fluid dynamics equations for each point
❏ Weather prediction (7 days in 24 hours): 56 Gflops
❏ Weather prediction (50 years in 30 days): 4.8 Tflops

8
Define: Distributed Systems

A distributed system is: “A collection of independent computers that appears to

its users as a single coherent system”
OR

A distributed system consisters of multiple autonomous computers that

communicate through a computer network

With a purpose to share resources such as; S/W, H/W and Data through middle ware
9
 Distributed Systems!
Cluster Computing

Cluster computing Grid computing

Nodes homogenous (HW/SW) e.g. OS Homogenous or heterogeneous
Computers are dedicated to specific task Computers in a grid contribute unused processing resources to
network
Connected by high speed LAB, BUS Through low speed BUS or internet
Centralized network topology De-centralized network topology
Work as a single system Every node is autonomous any node can opt out any time 10
Distributed Systems!

11
Distributed Systems!

12
Atomic Broadcast

❏ Atomic Broadcast – The same set of messages is

delivered to all the processes in the same order
❏ A node broadcast a message
❏ If sender correct, all correct nodes deliver msg
GAB GAB GAB
❏ All correct nodes deliver same msg LLT LLT LLT
❏ Messages delivered in the same order
❏ Group membership services and atomic broadcast (GAB)
is responsible for the cluster membership and
communication functions
❏ LLT sending information GAB maintains cluster membership by
❏ GAB make that sending/ communication easy (through GAB ports) receiving heartbeat messages from
each node by LLT.
13
Atomic broadcast - Consensus

❏ Consensus – All processes decide upon one common value among those
proposed
❏ Given atomic broadcast

❏ Can You Solve CONSENSUS problem?

14
 Distributed OPERATING System

❏ To support heterogeneous computers and

network to build a single view
❏ Distributed systems organized by means
of a layer
❏ Placed between users app and OS
❏ Such distributed systems are called
middleware

15
Goals of Distributed Systems

❏ Four important goals to meet to build a distributed system

❏ Make resource available
❏ Distribution transparency
❏ Scalability
❏ Pitfalls

16
1. Make resources Available

❏ Main goal is make it easier for the user to access/share remote resources
❏ resource can be anything
❏ printer, computers, storage facilities, network etc.
❏ Lead to Security issues

17
2. Transparency

Definition of transparency is “Hide the fact that its processes and resources are
physically distributed”

There are different kinds of transparency exist in distributed system

Can you suggest any?

18
2. Transparency

❏ Access Transparency: Client should be unaware of the distribution of the

files, and how these files can be accessed - differences in machine
architectures
❏ Location Transparency: Client should be unaware of the physical location of
resources
http://www.prenhall.com/index.html

19
2. Transparency

❏ Migration Transparency: In distributed systems in which resources can be

moved without effecting how these resources can be accessed

❏ Relocation Transparency: In distributed systems in which resources can be

relocated while they are being accessed without user noticing anything

20
2. Transparency

❏ Replication Transparency: Resources are replicated to increase availability

and performance
Replication is hiding the fact that several copies of a resource exist
❏ Concurrency Transparency: Users and applications should be able to access
shared resources without interference between each other
lead to a consistency issues

21
2. Transparency

❏ Failure Transparency: The distributed system are prone to failures

Failure transparency is user does not notice that the resource fails to work
and that the system subsequently recover from the failure

22
Transparency Description

Access Hide differences in data distribution and how a resource is

accessed

Location Hide where a resource is located

Migration Hide that a resource may move to another location

Relocation Hide that a resource may be moved to another location while in

use

Replication Hide that a resource may be shared by several competitive users

Concurrency Hide that a resource may be shared by several competitive users

Failure Hide the failure and recovery of a resource

23
Degree of Transparency

❏ Hide distribution aspects is not a good idea?

24
Degree of Transparency

❏ There is a trade-off between degree of transparency and performance

❏ Communication among processes - Network delay
❏ Internet applications repeatedly try to contact server before trying another
and finally giving up
❏ Replicas located on different continents, need to consistent, change in one
requires seconds to update all
❏ Better to print job to a busy nearby computer instead of ideal one at
corporate headquarters in a different country
25
Openness

Another important goal of distributed systems

❏ An open distributed system is a system that offers services according to

standard rules
Interoperability - Two implementation of a system from different
manufacturers can work together
Portability- Application developed by distributed system A can be executed
without modification on system B

26
3. Scalability

Measured along at least three different dimensions

❏ Size scalability
❏ Geographically scalability
❏ Administratively scalability

Most systems account only, to a certain extent, for size scalability. The
(non)solution: powerful servers.

Today, the challenge lies in geographical and administrative scalability 27

3. Scalability

The server becomes a bottleneck as the number of users grows

using only a single server is sometimes unavoidable

❏ centralized services
❏ centralized data
❏ centralized algorithms

28
3. Scalability

Geographical scalability

❏ Mostly distributed systems are designed for LAN

❏ for synchronous communication
❏ block until receives a response
❏ communication is unreliable
❏ broadcast message to discover service

29
3. Scalability

Administrative Scalability

❏ scalability among different administrative domains

❏ they may have different policies
❏ Resource usage
❏ Management
❏ Payment management and
❏ Security

30
 4. Pitfalls

Peter Deutsch (Sun microsystem) formulated mistakes as a false assumption

developers makes

❏ network is reliable
❏ network is secure
❏ network is homogenous - topology does not change
❏ latency is zero
❏ bandwidth is infinite
❏ transport cost is zero
❏ there is one administrator

31
Types of Distributed Systems

Various types of distributed systems

❏ Distributed Computing System

❏ Distributed Information System
❏ Distributed Embedded System

32
1. Distributed Computing System - Grid System

“A Grid computing System is a collection of distributed computing available over

a local or wide area network, that appears to an end user or application as one
large virtual computing system”

It is an approach that spans not only location but also organizations, and machine
architectures.

Internet – getting computers to talk together

Grid Computing – getting computers work together 33

1. Distributed Computing System - Grid System

❏ collection of computers running the same operating system OR as complex

as systems comprised of different OS
❏ server, which handles all the administrative duties for the system (control
node, dispatcher)
❏ Nodes running special grid computing network software - middleware
❏ Grid middleware: to run a process or application across the entire network
of machines.
❏ Middleware is the workhorse of the grid computing system
❏ Lack of standards 34
1. Distributed Computing System - Grid System

❏ Control node – dispatcher

❏ Scheduling/priority task
❏ Monitor systems
❏ Resource allocation
❏ Grid middleware
❏ Process launch
❏ communicate

35
1. Distributed Computing System - Grid System

Applications of Grid - Screensaver is a Lifesaver

❏ Search for Extra-terrestrial Intelligence (SETI)

❏ SETI project is to analyze data gathered by radio telescopes
❏ search of evidence for intelligent alien communications
❏ There's far too much information for a single computer to analyze
effectively
❏ Folding@home project - Stanford University's chemistry department
❏ proteins take certain shapes - Scientists believe that protein
36
1. Distributed Computing System - Cluster
Computing
❏ Collection of systems that work together, can be viewed as a single
computer
❏ Underlying hardware consists of collection of similar PCs
❏ Connected with high speed networks
❏ Each node run the same OS (linux)
❏ Defination of cluster is extend further
❏ HA (High Availability Cluster)
❏ LB (Load-balancing Cluster) 37
38
Load Balancer

39
1. Distributed
Computing
System - Cluster
Computing

40
2. Distributed Information Systems

❏ Typical system includes a database

❏ Integration of such system is quite difficult
❏ Client can wrap number of request into single request and have it executed
as a distributed transaction
❏ Interoperability is a painful process
❏ Transaction Processing system
❏ Enterprise Application Integration

41
2. Distributed Information Systems

Transaction processing system

42
 2. Distributed Information Systems

❏ Enterprise Application Integration

❏ Publish/subscribe model

43
 3. Distributed Pervasive Computing
❏ Mobile and embedded computing
devices
❏ Instability is the default behavior
❏ Being small, battery operated having
wireless connections
❏ Lack of administrative control
❏ (Grimm et al. 2004 )Features of
Pervasive applications
❏ Embrace contextual changes
❏ Encourage ad hoc composition
44
❏ Recognize sharing as default
 3. Distributed Pervasive Computing

❏ Distributed Home Systems

❏ Popular type of pervasive system
❏ Comprises of TV, audio, video equipment,
game devices, PDA’s as a single system
❏ Challenges
❏ Self-configuring, self-managing
❏ Achieved through UPnP standards – obtain
IP address
45
 3. Distributed Pervasive Computing

❏ Electronic Health Care Systems

❏ System equipped with sensors
organized in BAN
❏ No strings attached to immobile
device
❏ Minimal hinder a person

46