Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
4 views

Lecture 1 Introduction

The document outlines the CSC334 course on Parallel & Distributed Computing, covering key concepts such as parallel and distributed computing, application profiling, and programming with OpenMP and MPI. It highlights prerequisites, recommended materials, and the significance of high-performance computing (HPC) and GPU-accelerated computing. Additionally, it discusses various types of parallel and distributed computing, emphasizing the advantages of parallel computing in solving complex problems and improving efficiency.

Uploaded by

zaeemrana69
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
4 views

Lecture 1 Introduction

The document outlines the CSC334 course on Parallel & Distributed Computing, covering key concepts such as parallel and distributed computing, application profiling, and programming with OpenMP and MPI. It highlights prerequisites, recommended materials, and the significance of high-performance computing (HPC) and GPU-accelerated computing. Additionally, it discusses various types of parallel and distributed computing, emphasizing the advantages of parallel computing in solving complex problems and improving efficiency.

Uploaded by

zaeemrana69
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 34

CSC334 Parallel & Distributed Computing

Lecture # 01
Introduction
• Dr. Samera Batool
• Samera.Batool@comsats.edu.pk
• Ground Flood Room No G41
Course Overview
• This course covers following main concepts
• Concepts of parallel and distributed computing
• Analysis and profiling of applications
• Shared memory concepts
• Distributed memory concepts
• Parallel and distributed programming (OpenMP, MPI)
• GPU based computing and programming (CUDA)
• Virtualization
• Cloud Computing, MapReduce
• Grid Computing
• Peer-to-Peer Computing
• Future trends in computing
Course Pre-requisites
• Programming Experience (preferably Python/C++/Java)
• Understanding of Computer Organization and Architecture
• Understanding of Operating System
Recommended Material
• Distributed Systems, Maarten van Steen & Andrew S. Tanenbaum, 3rd Edition (2020),
Pearson.
• Parallel Programming: Concepts and Practice, Bertil Schmidt, Jorge Gonzalez-
Dominguez, Christian Hundt, Moritz Schlarb, 1st Edition (2018), Elsevier.
• Parallel and High-Performance Computing, Robert Robey and Yuliana Zamora, 1st
Edition (2021).
• Distributed and Cloud Computing: From Parallel Processing to the Internet of Things,
Kai Hwang, Jack Dongarra, Geoffrey Fox, 1st Edition (2012), Elsevier.
• Multicore and GPU Programming: An Integrated Approach, Gerassimos Barlas, 2nd
Edition (2015), Elsevier.
• Parallel programming: For multicore and cluster systems. Rauber, Thomas, and
Gudula Rünger. Springer Science & Business Media, 2013.
Single Processor Architecture
Memory Hierarchy
Application
Partitioning
High-Performance Computing (HPC)
• HPC is the use of parallel processing for running advanced application
programs efficiently, reliably and quickly.
• It applies especially to systems that function above a tera FLOPs
(floating-point operations per second) processing speed.
• The term HPC is occasionally used as a synonym for supercomputing,
although technically a supercomputer is a system that performs at or
near the currently highest operational rate for computers.
High Performance Computing
GPU-accelerated Computing
• GPU-accelerated computing is the use of a graphics processing unit
(GPU) together with a CPU to accelerate deep learning, analytics, and
engineering applications.
• Pioneered in 2007 by NVIDIA, GPU accelerators now power energy-
efficient data centers in government labs, universities, enterprises,
and small-and-medium businesses around the world.
• They play a huge role in accelerating applications in platforms ranging
from artificial intelligence to cars, drones, and robots.
What is GPU?
• It is a processor optimized for 2D/3D graphics, video, visual
computing, and display.
• It is highly parallel, highly multithreaded multiprocessor optimized for
visual computing.
• It provide real-time visual interaction with computed objects via
graphics images, and video.
• It serves as both a programmable graphics processor and a scalable
parallel computing platform.
• Heterogeneous Systems: combine a GPU with a CPU
HPC System
composition
Parallel Computers
• Virtually all stand-alone computers
today are parallel from hardware
perspective:
• Multiple functional units (L1 cache, L2
cache, branch, pre-fetch, decode,
floating-point, graphics processing
(GPU), integer, etc.)
• Multiple execution units/cores
• Multiple hardware threads

IBM BG/Q Compute Chip with 18 cores (PU) and 16 L2 Cache units (L2)
Parallel Computers
• Networks connect multiple stand-
alone computers (nodes) to make
larger parallel computer clusters.
• Parallel computer cluster
• Each compute node is a multi-
processor parallel computer in itself
• Multiple compute nodes are
networked together with an
Infiniband network
• Special purpose nodes, also multi-
processor, are used for other
purposes
Types of Parallel and Distributed
Computing
• Parallel Computing
• Shared Memory
• Distributed Memory

• Distributed Computing
• Cluster Computing
• Grid Computing
• Cloud Computing
• Distributed Pervasive Systems
Parallel Computing
Distributed (Cluster) Computing
• Essentially a group of high-end
systems connected through a LAN
• Homogeneous: same OS, near-
identical hardware
• Single managing node
Distributed (Grid) Computing
• Lots of nodes from everywhere
• Heterogeneous
• Dispersed across several organizations
• Can easily span a wide-area network

• To allow for collaborations, grids generally use virtual organizations.


• In essence, this is a grouping of users (or their IDs) that will allow for
authorization on resource allocation.
Distributed (Cloud) Computing
Distributed (Pervasive) Computing
• Emerging next-generation of distributed systems in which nodes are
small, mobile, and often embedded in a larger system, characterized
by the fact that the system naturally blends into the user’s
environment.
• Three subtypes
• Ubiquitous computing systems: pervasive and continuously present, i.e.,
there is a continuous interaction between system and user.
• Mobile computing systems: pervasive, but emphasis is on the fact that
devices are inherently mobile.
• Sensor (and actuator) networks: pervasive, with emphasis on the actual
(collaborative) sensing and actuation of the environment.
Why Use Parallel
Computing?
The Real World is Massively
Parallel
• In the natural world, many
complex, interrelated events are
happening at the same time, yet
within a temporal sequence.
• Compared to serial computing,
parallel computing is much better
suited for modeling, simulating
and understanding complex, real
world phenomena.
• For example, imagine modeling
these serially =>
SAVE TIME AND/OR MONEY
(Main Reasons)
• In theory, throwing more
resources at a task will shorten
its time to completion, with
potential cost savings.
• Parallel computers can be built
from cheap, commodity
components.
SOLVE LARGER / MORE COMPLEX PROBLEMS
(Main Reasons)
• Many problems are so large and/or complex that it is impractical or
impossible to solve them on a single computer, especially given
limited computer memory.
• Example: Web search engines/databases processing millions of
transactions every second
PROVIDE CONCURRENCY
(Main Reasons)
• A single compute resource can only do one thing at a time. Multiple
compute resources can do many things simultaneously.
• Example: Collaborative Networks provide a global venue where
people from around the world can meet and conduct work "virtually".
MAKE BETTER USE OF UNDERLYING PARALLEL
HARDWARE
(Main Reasons)
• Modern computers, even laptops,
are parallel in architecture with
multiple processors/cores.
• Parallel software is specifically
intended for parallel hardware
with multiple cores, threads, etc.
• In most cases, serial programs run
on modern computers "waste" Intel Xeon processor with 6 cores and 6 L3
potential computing power. cache units
The Future
(Main Reasons)
• During the past 20+ years, the trends
indicated by ever faster networks,
distributed systems, and multi-
processor computer architectures
(even at the desktop level) clearly
show that parallelism is the future of
computing.
• In this same time period, there has
been a greater than 500,000x increase
in supercomputer performance, with
no end currently in sight.
•

You might also like