Introduction to Distributed and Parallel

Computing
CS-401

Dr. Sanjay Saxena

Visiting Faculty, CSE, IIIT Vadodara
Assistant Professor, CSE, IIIT Bhubaneswar
Post doc – University of Pennsylvania, USA
PhD – Indian Institute of Technology(BHU), Varanasi
 What is Distributed Systems/Computing?
• Simply, Distributed System is a group of computers working together as to appear as a single computer to the end-user.

• Distributed computing refers to a system where processing and data storage is distributed across multiple devices or
systems, rather than being handled by a single central device.

These machines have a Distributed computing

shared state, operate refers to the use of
concurrently and can fail distributed systems to
independently without complete computing
affecting the whole tasks.
system’s uptime.
Contd..
 Types of distributed systems
Client-Server — In this client sends requests to the server then, the server responds to the client requests. Clients contact
the server for data, then format it and display it to the end-user. The end-user can also make changes from the client-side and
commit them back to the server to make it permanent. There are three main methods — sockets, remote procedure calls,
pipes.
 Contd..
Three-tier — It helps move the client intelligence to a middle-tier to use stateless clients. The client doesn’t directly
communicate with the database server. The client interacts with the server, which in turn interacts with the database. It
simplifies application deployment. Most web applications are three-tier.
 Peer-to-peer
There are no special machines that provide a service or manage the resources. All of the responsibilities are
uniformly distributed among all the machines (peers). They serve both as a client and as a server. Torrent is a good
example of a peer-to-peer network.
Different Components Characteristics
• Devices or Systems: The devices or systems in a • Multiple Devices or Systems: Processing and data storage is

distributed system have their own processing capabilities distributed across multiple devices or systems.

and may also store and manage their own data. • Peer-to-Peer Architecture: Devices or systems in a

• Network: The network connects the devices or systems distributed system can act as both clients and servers, as they

in the distributed system, allowing them to communicate can both request and provide services to other devices or

and exchange data. systems in the network.

• Resource Management: Distributed systems often have • Shared Resources: Resources such as computing power,

some type of resource management system in place to storage, and networking are shared among the devices or

allocate and manage shared resources such as computing systems in the network.

power, storage, and networking. • Horizontal Scaling: Scaling a distributed computing system
typically involves adding more devices or systems to the
network to increase processing and storage capacity. This can be
done through hardware upgrades or by adding additional devices
or systems to the network..
 Applications Area of Distributed System

Finance and Commerce: Amazon, eBay, Online Banking, E-Commerce websites.

Information Society: Search Engines, Wikipedia, Social Networking, Cloud Computing.
Cloud Technologies: AWS, Salesforce, Microsoft Azure, SAP.
Entertainment: Online Gaming, Music, youtube.
Healthcare: Online patient records, Health Informatics.
Education: E-learning.
Transport and logistics: GPS, Google Maps.
Environment Management: Sensor technologies.
 Advantages and Disadvantages
Scalability: Distributed systems are generally more scalable than centralized systems, as they can easily add
new devices or systems to the network to increase processing and storage capacity.
Reliability: Distributed systems are often more reliable than centralized systems, as they can continue to
operate even if one device or system fails.
Flexibility: Distributed systems are generally more flexible than centralized systems, as they can be
configured and reconfigured more easily to meet changing computing needs.
There are a few limitations to Distributed Computing System
Complexity: Distributed systems can be more complex than centralized systems, as they involve multiple
devices or systems that need to be coordinated and managed.
Security: It can be more challenging to secure a distributed system, as security measures must be
implemented on each device or system to ensure the security of the entire system.
Performance: Distributed systems may not offer the same level of performance as centralized systems, as
processing and data storage is distributed across multiple devices or systems.
Significant Applications
Distributed Computing Systems have a number of applications, including:
Cloud Computing: Cloud Computing systems are a type of distributed computing system that are used to deliver resources
such as computing power, storage, and networking over the Internet.
Peer-to-Peer Networks: Peer-to-Peer Networks are a type of distributed computing system that is used to share resources
such as files and computing power among users.
Distributed Architectures: Many modern computing systems, such as microservices architectures, use distributed
architectures to distribute processing and data storage across multiple devices or systems.
Cloud Computing

Simply, Cloud computing is

the delivery of computing
services—including servers,
storage, databases, networking,
software, analytics, and
intelligence—over the Internet
(“the cloud”) to offer faster
innovation, flexible resources,
and economies of scale.
You may not realize it but we have been using cloud computing
for over a decade!

Ever sent an email with an attachment to yourself from your home computer, and then signed in to your email at work
to access the attachment?
One of the earliest forms of cloud computing was the ability to save attachments in your email inbox and access
them on another computer. That's basic cloud storage for you!
 Cloud Service Models
Cloud solutions come in three primary service models: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and
Software as a Service (SaaS).
Real World
Example
 PaaS
This service layer is primarily
IaaS geared towards developers and
IaaS gives users access to storage, networking, operations professionals. Service
servers, and other computing resources via the providers rent out cloud-based
cloud. platforms for users to develop and
For example, many businesses use IaaS to deliver applications.
support workload spikes during busy seasons
(like holidays). Saas
Popular IaaS providers include: Cloud application services are the most
Amazon Web Services (AWS), Microsoft Azure,
well-known of the cloud service
Google Compute Engine (GCE), the IaaS models. The software is hosted,
component of Google Cloud Platform (GCP)
packaged, and delivered by a third
party through the Internet (typically on
a browser-based interface).
 Benefits of Cloud Computing

Fewer Expenses
Moving to the Cloud will reduce the cost of IT systems management and maintenance. Instead of purchasing new systems
and equipment for your business, you can take advantage of the resources from your service provider and reduce the cost
of system upgrades, energy consumption, and labor.
Security
Most providers follow policies to ensure the security and protection of your data and infrastructure against threats. Also, it
enables data backup and recovery which can be reflected throughout multiple services and can be accessed at any time.
Flexibility
We all know how important it is for businesses to be flexible especially in these trying times. Cloud computing will allow
your employees to work anywhere since they will get access to data, provided they have an internet connection.
Collaboration
When your team needs to work on a certain project, and each are in different locations, they can utilize cloud computing to
work and share files at the same time. It is secure and will allow your business to connect and communicate easily.
Automatic Updates
Cloud-based applications automatically update so you don’t need to call for support to manually install updates on your
device every time. Updates could include the latest versions in addition to server upgrades and more, which allows your
business to save time and money.
 Parallel Computing
Parallel computing refers to the process of executing several processors an application or computation simultaneously.

Advantages It saves time and money as many resources working

together will reduce the time and cut potential costs.
1.It can be impractical to solve larger problems on Serial
Computing.
2.It can take advantage of non-local resources when the local
resources are finite.
3.Serial Computing ‘wastes’ the potential computing power, thus
Parallel Computing makes better work of the hardware.
Difference between Distributed computing and Parallel
Computing

There are two types of computations: parallel computing and distributed computing. Parallel computing allows several
processors to accomplish their tasks at the same time. In contrast, distributed computing splits a single task among numerous
systems to achieve a common goal.
 Types of Parallelism
Ø Bit-level parallelism –
It is the form of parallel computing which is based on the increasing processor’s size. It reduces the number of instructions
that the system must execute in order to perform a task on large-sized data.
Example: Consider a scenario where an 8-bit processor must compute the sum of two 16-bit integers. It must first sum up the
8 lower-order bits, then add the 8 higher-order bits, thus requiring two instructions to perform the operation. A 16-bit
processor can perform the operation with just one instruction.

Ø Instruction-level parallelism –
A processor can only address less than one instruction for each clock cycle phase. These instructions can be re-ordered and
grouped which are later on executed concurrently without affecting the result of the program. This is called instruction-level
parallelism.

Ø Task Parallelism –
Task parallelism employs the decomposition of a task into subtasks and then allocating each of the subtasks for execution.
The processors perform the execution of sub-tasks concurrently.

Ø Data-level parallelism (DLP) –

Instructions from a single stream operate concurrently on several data – Limited by non-regular data manipulation patterns
and by memory bandwidth
 Why parallel computing?
Ø The whole real-world runs in dynamic nature i.e. many things happen at a certain time but at different
places concurrently. This data is extensively huge to manage.
Ø Real-world data needs more dynamic simulation and modeling, and for achieving the same, parallel
computing is the key.
Ø Parallel computing provides concurrency and saves time and money.
Ø Complex, large datasets, and their management can be organized only and only using parallel
computing’s approach.
Ø Ensures the effective utilization of the resources. The hardware is guaranteed to be used effectively
whereas in serial computation only some part of the hardware was used and the rest rendered idle.
Ø Also, it is impractical to implement real-time systems using serial computing.
Applications of Parallel Computing:
•Databases and Data mining.
•Real-time simulation of systems.
•Science and Engineering.
•Advanced graphics, augmented reality, and virtual reality.
Limitations of Parallel Computing:
•It addresses such as communication and synchronization between multiple sub-tasks and processes which is difficult to
achieve.
•The algorithms must be managed in such a way that they can be handled in a parallel mechanism.
•The algorithms or programs must have low coupling and high cohesion. But it’s difficult to create such programs.
•More technically skilled and expert programmers can code a parallelism-based program well.
Thanks & Cheers!!

Small aim is a crime; have great aim.

Bharat-Ratan A. P. J. Abdul Kalam