Cloud&Distributed Computing

An Overview of Cloud & Distributed Computing -
Programming Paradigms
VCV.Rao
Centre for Development of Advanced Computing (C-DAC),
Pune University Campus
National Workshop on Big Data Analytics (BiDA2014) at CRRao AIMSCS

August 22-24, 2014,
jointly with CMSD, U of Hyderabad, & Computer Society of India.
VCV.Rao. C-DAC, Pune
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
VCV. Rao, Ph.D

Associate Director /HoD
HPC Frontier technologies Exploration (HPC-FTE) Group
Centre for Development of Advanced Computing (C-DAC)
Pune University Campus,
GaneshKhind, Pune
Maharashtra State, India
Email : vcvrao@cdac.in
CDAC is a R&D Institute of Department of Electronics and Information Technology (DeitY) ,
Ministry of Communications & Information Technology (MCIT) );
Government of India
Distributed and Cloud Computing Prog. & Software Environments An Overview

Lecture Outline
Following topics will be discussed

An Overview of Distributed Computing Architectures
(HPC Systems High Performance Computing, & High
Throughput Computing, Grid Computing & Utility
Computing) and Cloud Computing
Programming on Cloud and Distributed Computing
Systems
Opportunities for Applications
References & Source for Presentation

The Presentation is prepared based on Authors experience on
various research projects on Cloud & Distributed Computing as
well as references given in this presentation.
Source :
Text Books, Research Articles, Web Sites as indicated in many
slides and References of this presentation
Courtesy :
Authors research work as indicated in Text Books, Research
Articles, Web Sites as indicated in many slides and References of
this presentation
C-DAC , in brief
Centre for Development of Advanced Computing (C-DAC)

is the
premier R&D organization of the Department of
Electronics and Information Technology (DeitY), Ministry of
Communications & Information Technology (MCIT) for

carrying out R&D in IT, Electronics and associated areas.
The setting up in 1988 - C-DAC has been undertaking

building of multiple generations of Supercomputer starting
from PARAM with 1 GF in 1988.
Distributed System Models & Enabling Technologies
Scalable Computing over the Internet

The New Computing Paradigms
Computing Paradigm Distinctions
Centralized Computing
Parallel Computing
Distributed Computing
Cloud Computing
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
Distributed System Models and Enabling Technologies

High-Performance Computing
Supercomputers (massively parallel processors or MPPs)

are
gradually
replaced
by
clusters
of
cooperative
computers out of desire to share computing resources).

The Cluster is often a collection
of homogeneous
computer nodes that are physically connected in close

range to one another.

High-Throughput Computing (HTC)
Peer-to-peer (P2P) networks are formed for distributed file

sharing and content delivery applications.
Change will take place on HPC Paradigms to an HTC

paradigms (High-flux computing). The applications for highflux computing is the Internet searches and web services by
million or more users simultaneously.
Distributed Comp. & Software Environments

Parallel and Distributed Programming Paradigms
Consider a distributed computing system consisting of a set of network nodes
or workers. For Parallelization of application on Distributed Cloud computing,
below important issues should be addressed.
Partitioning
Computation Partitioning
Data Partitioning
Mapping
Synchronization
Communication
Scheduling
Source : Distributed and Cloud Computing Book, From Parallel Processing to

the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann,
Publishers 2012
9
Emerging Parallel Processing Platforms: Challenges

MIMD : Parallel Processing Platforms Software & Hardware
Scalability Vs. Single System Image

Distributed
Computer
Systems
Better Performance
Clusters
SMPs
Scalability Vs. Single System Image

MPPs
Future Cluster
Accelerators
Distributed
System
Single-System Image
Overlapped design space of
Clusters, MPPs, SMPs, and
Distributed Computer systems
MPP
CC-NUMA
Present
Cluster
SMP
UP
Single System Image

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
10
Shared Pool of Computing Applications

Shared Pool of Computing
Resources: Processors,
Memory, Disks.
Guarantee at least one

workstation to many
individuals (when active)
Deliver large % of collective

resources to few individuals at
any one time.
11
Emerging Parallel Processing Platforms: Challenges

Current Parallel Processing Platforms
Cray XT4, XT5; IBM Systems
CrayX1EVector Processing
One-to-All Broadcast
One-to-All Personalized
Communication
All-to-All Broadcast
Cluster of Multi-Core Systems

PARAM Series
All-to-All personalized
Communication
Circular Shift
ANUPAM
Reduction
ANURAG
Prefix Sum
3D torus Topology
NETWORK
Multi Core Node 0
CPU0
CPU1
CPU2
Memory
Multi Core Node 1

CPU3
CPU0
CPU1
CPU2
CPU3
Memory
Cluster of Multi Core Processor Systems

Source : References
12
Emerging Parallel Processing Platforms: Clusters

Future/ Idealized Cluster Architecture
Programming Environment : GLSL(OpenGL Shading Language);
HLSL(high level shader language); Direct X, Cg, OpenCL, OpenMP
Programming Environment
(Java, C, Fortran, MPI, PVM)
top-level switches (288 ports)
Connects to 12 nodes
node-level switches (24 ports)
Web Windows Other Subsystems

User Interface (Database, OLTP)
NVIDIA-Kepler-CUDA
Single System Image Infrastructure

Availability Infrastructure
AMD Fire Stream
User
Application
OS
OS
Node
Node
MPI
OS
TCP
IP
Node
Intel Xeom-Phi
Driver
NIC
Hardware
Gigabit Commodity Interconnect/SAN
C-DAC RC-FPGA
13

High
(100%)
Grid
Cluster
System availability
MPP
Cloud
NUMA
P2P
network
SMP
Low
(0)
Small
System size (# processor cores)
Large (106)
Estimated system availability by system size of common configurations in 2010.

by Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
14
Scalability (No. of processors or

cores in a system)
107
106
P2P
105
Cluster
104
Cloud
103
NUMA
Grid
102
10
SMP
1
1
10
102
103
104
105
106
107
System scalability versus multiplicity of OS images

by Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
15
Applications in Grid Computing

Classification of Grid applications
Grid aware applications
Multi-disciplinary Applications
All applications which

make use of coupled
computational resources
that are not available at a
SINGLE SITE
SAME
Grid Services: A Layered Grid Architecture

Coordinating multiple resources:
ubiquitous infrastructure services, appspecific distributed services
Sharing single resources: negotiating
access, controlling use
Talking to things: communication
(Internet protocols) & security
Application
Collective
Application
Resource
Connectivity
Controlling things locally: Access

to, & control of, resources
Transport
Internet
Fabric
Source : References & The Globus CoG Home Page. http://www.globus.org/cog.

Globus. http://www.globus.org. http://www.teragrid.org/
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov. http://www.cdac.in/
Internet Protocol Architecture
Meta Applications
Link
16
Applications in Grid Computing
Compute hardware
Intel/Linux Clusters, Alpha SMP clusters,
POWER4 cluster,
Large-scale storage systems
hundreds of terabytes for secondary storage
Very high-speed network backbone
bandwidth for rich interaction and tight
coupling
Grid middleware
Globus, data management,
Next-generation applications
Grid Systems
Grids are distributed systems that enable

the sharing, selection, and aggregation of
resources distributed across "multiple"
administrative domains based on
availability, capability, performance, cost,
and quality-of-service requirements
Categories of Grid Applications

First-in-time, or an on-demand
computing
Data-Intensive Computing
Collaborative Computing
High Throughput Computing
Distributed Supercomputing
Comp. Grid
Distributed Super
Computing
High Throughput
Data Grid
On Demand
Service Grid
Grid Taxonomy
Collaborative
Multi-Media

Globus. http://www.globus.org. http://www.teragrid.org/
17
Utility Computing
Utility computing is the packaging of Computing resources such as computation
and storage, as a metered service similar to a traditional Public Utility(such as
electricity, water, natural gas or telephone network ).
More related to cloud computing
Amazon Web Service (AWS)
Applications, storage, computing power and
Elastics Computer Cloud (EC2)
network
Simple Storage Service (S3)
Requires cloud like infrastructure
Pay by the drink model
EMC cloud Storage
Similar to electric service at home
Microsoft Azure
Pay for extra resources when needed
Google App Engine
To handle expected surge in demand
Unanticipated surges in demand
Better economics
A utility computing service is one in which customers receive computing resources

from a service provider (hardware and/or software) and pay by the drink, much
as you do for your electric service at home
Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley & Sons, 2009.
18

The trend towards Utility Computing: The vision of computer utilities in
modern distributed computing systems.
Web services
Data centers
Utility computing
Technology
Service
computing
convergence
HTC in business and HPC

in scientific applications
Grid computing
P2P computing
Cloud computing
Computing paradigms
Attributes/capabilities
Ubiquitous: Reliable and scalable

Automatic: Dynamic and discovery
Composable: QoS, SLA, etc.
Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley & Sons, 2009.
19
Distributed and Cloud Computing

Disparate nodes
HTC Systems
HPC Systems
High speed
File sharing
Distributed control
Homogenous nodes
P2P network
Clusters of MPPs
Centralized control
Disparate clusters
Geographically sparse
Computational and data grids
Service-oriented architecture (SOA)
RFID and sensors

Virtualization
Web 2.0 services
Internet of Things
Internet clouds
Evolutionary trend toward parallel distributed, and cloud computing with clusters,
MPPs, P2P network and grids and clouds, web services, and the Internet of Things.
20
Extreme-Scale Computing Challenges

HPC Accelerators
Low Power
Consumption
GPU
(Nvidia, AMD)
FPGA
MIC
(Xilinx, Altera) (Intel Xeon Phi)
Effort to port and optimize code for a accelerator type
Speed versus Efforts : Speedup over CPU vs Time to develop

for a particular accelerator type
Application Topologies /Network
Courtesy & Source : Reference : Intel, AMD, NVIDIA, Xlinx, Altera;
David Bader (References - 176-179)
21
Accelerator Comp.: Prog. Env. : Energy Efficiency

Shared Address Space Programming (Offload, Native, Host)
OpenMP, Intel TBB, Cilk Plus, Pthreads
Shared Address Space Programming (Only Offload)

OpenMP , Intel TBB, Pthreads - CUDA
Message Passing Programming (MPI)

(Offload MIC Offload /Host Offload)
MPI - (Symmetric & Co-processor /Host)
Message Passing Programming (MPI)

(Offload MIC Offload /Host Offload)
MPI - (Symmetric & Co-processor /Host)
Hybrid Programming
(MPI OpenMP, MPI Cilk Plus
MPI-Intel TBB, MPI-Pthreads)
Application
Host
Coprocessor
Application
Host
Coprocessor
Hybrid Programming (MPI CUDA)

Source : References & http://www.intel.com/; http://www.nvidia.com, http://www.adm.com, http://www.cdac.in
22
Applications
Software Threading
Multi-to-May-Core Systems UMA & NUMA Systems
Hybrid Computing
Prog.API - Multi-Core Systems with Devices
Very-high
level
Computational
Science
Data
Informatics
Information
Technology
Efficient, Deterministic, Declarative, Restrictive

Expressiveness based language
High level
Parallel Prog. Languages (OpenMP, PGAS, Intel TBB,

Cilk Plus, OpenACC, CUDA)
Low-level
Low-level APIs (MPI, Pthreads, OpenCL, Verilog)
Very
low-level
Machine code, Assembly
Heterogeneous
Hardware
Coprocessors
GPUs
FPGAs
Source : NVIDIA, AMD, SGI, Intel, IBM Alter, Xilinux & References
23
Cloud Computing over the Internet

Paid services
Hardware
Software
Internet cloud
User
Storage
Network
Submit requests
Service
Virtualized resources from data centres to form a Internet cloud,

provisioned with hardware, software, storage, network, and
services for paid users to run their applications
24
Cloud Computing over the Internet

Features of Cloud & Grid Platforms
Cloud Capabilities & Platform
Features
Traditional Features Common to
Grids and Clouds
Data Features and Databases
Programming and Runtime Support
Paid
services
User
Submit
Cloud Platforms Capabilities

requests
Should offer cost-effective utility
computing with the elasticity to scaleup and down in power
Commercial Clouds offer different
capabilities
Commonly termed as Platform as a
Service (PaaS) Ex : Azure
Commonly termed as Infrastructure
as a service (Iaas) Ex : Amazon
Hardware
Software
Internet cloud
Storage
Network
Service
25
Cloud Computing and Service Models
Public, Private and Hybrid Clouds

A typical public cloud
Data center
Cloud service
queues
Server cluster (VMs)
Platform fronted
(web service API)
To users or other
public cloud over the
Internet
Cloud storage
Amazon
AWS
Microsoft
Azure
IBM Blue
Cloud
The Internet
A hybrid cloud
Private cloud
(IBM RC2)
Public
Salesforce
Force.com
Google App
engine
...
An
Intranet
Cloud users
Public, private and hybrid clouds illustrated by functional architecture and

connectivity of representative clouds
26
Cloud Computing and Service Models

Data-Center Networking Structure
Internet
Internet
BR
Data Center
Layer 3
BR
AR
AR
LB
LB
AR
AR
Layer 2
Key:
BR = L3 border router
AR = L3 access router
S = L2 switch
LB = Load balancer
A = Rack of servers
A single layer 2 domain

Standard data-center networking for the cloud to access the Internet.
27
Cloud Programming & Software Environment

Parallel Computing and Programming Paradigms
MapReduce, Twister and Iterative MapReduce
Hadoop Library from Apache
Dryad and DryadLINQ from Microsoft
Sawzail and Pig Latin High-Level Languages
Mapping Apps. to Parallel & Distributed Systems
Monitor & Manage

Resources
Programming on Amazon AWS and Micro Azure

Programming on Amazon EC2
Amazon Simple Storage Service (S3)
Amazon Elastic Block Store (EBS) and SimpleDB
Microsoft Azure Programming Support
HPC/HTC System
Infrastructure
Access
Services
Service Consumers
Cloud
Administrator
IT Cloud
Component
Library
Programming Support Google App Engine

Programming on Google App Engine
Google File System (GFS)
BigTable , Googles NOSQL System
Chubby, Googles Distributed Lock Service
Component Vendors /
Software Publishers
Publish & Update

Components
28
Distributed System Models & Virtualization
Technologies for Network-based systems
Virtual Networking and Virtualization Middleware

Virtual Machines (VMs) - X86 Architecture Guest OS
Between the VMs and host-platform, one need to deploy
a middleware layer called a virtual machine monitor
(VMM)
A native VM installed with the use of a VMM called a
hypervisor in privileged node.
29
Cloud Computing : Storage Virtualization

Virtual
Servers
Virtual
Storage
Logical
Representation
Virtual
Network
Virtual
Applications
Middleware
Virtual
Clients
Storage Virtualization is
Technology that makes one set of resources look and feel
like another set of resources, preferably with more
desirable characteristics
A logical representation of resources not constrained by
Physical limitations
Virtualization
- Hides some of the complexity

- Adds or integrates new function with existing services
- Can be nested or applied to multiple layers of a system
Physical Resources
Programming APIs Integrating with Schedulers & Resource Utilization

Threading on Multi-core CPUs & GPUs
30
Cloud Storage and Virtualization

Storage
Cloud storage is of two types : Object based & Block based
Block storage is like a raw disk space on which file system can be
created.
For both the storage types, backend can be anything like local attached
disks, nas, das, iscsi based or fiber channel based storage boxes.
In Cloud the use of these storage :
Mostly Object storage is used for holding Virtual Machine/Virtual
Instances templates & ISOs
This type can also be used to store the app/user related files.
Block storage serves the persistent disk space for Virtual
Machines/Virtual Instances.
The cloud giants, amazon also provides these options. S3

(Simple Storage Service; Object based storage) and elastic storage (block
based storage).
31

Storage
Cloud storage is of two types : Object based & Block based
Block storage is like a raw disk space on which file system can be created.
For both the storage types, backend can be anything like local attached disks, nas, das,
iscsi based or fiber channel based storage boxes.
In Cloud the use of these storage :

Mostly Object storage is used for holding Virtual Machine/Virtual Instances
templates & ISOs
This type can also be used to store the app/user related files.
Block storage serves the persistent disk space for Virtual Machines/Virtual
Instances.
The cloud giants, amazon also provides these options. S3

(Simple Storage Service; Object based storage) and elastic storage (block
based storage).
HPC Storage - Out-of-Core Algorithms Berkeley DB, Mango NO SQL,
HDF5.0 NetCDF
32
Storage : The opensource cloud middle-wares (OpenStack &

eucalyptus) For Openstack based implementation, refer these URLS:
High level overview of Object based (Swift) and block based
storage (Cinder)
http://www.openstack.org/software/openstack-storage/ [www.openstack.org]
Refer Storage Concepts in this url:

http://docs.openstack.org/admin-guide-cloud/content//ch_gettin>
cal-architecture [docs.openstack.org]
Swift (object based storage) concepts
http://docs.openstack.org/developer/swift/overview_architecture.html
[docs.openstack.org]
Info on cinder (block based storage) concepts & APIs
http://docs.openstack.org/developer/cinder [docs.openstack.org]
Service; Object based storage) and elastic storage (block based

storage).
33
Storage : Eucalyptus cloud based implementation, refer these urls :

Overview of object based storage (Walrus) in Eucalyptus clouds
: http://www.eucalyptus.com/blog/2013/03/18/eucalyptus-walrus-storageconsiderations [http://www.eucalyptus.com]
few slides on block storage (elastic block storage) in eucalyptus

cloud
: http://www.slideshare.net/eucalyptus/eucalyptus-elastic-block-storage-ebsin-devtest-scenarios
[www.slideshare.net]
APIs & documentation

: https://github.com/eucalyptus/eucalyptus/wiki/Storage [github.com]
Storage types in Eucalyptus

: http://www.eucalyptus.com/blog/2012/10/03/cloud-storage-types
[www.eucalyptus.com]
34
Programing on Cloud & Distributed Computing

Parallel and Distributed Programming Models & Tool Sets.
Model
Description
PVM
PVM is a software package that permits a heterogeneous collection

of serial, parallel and vector computers which are connected to a
network to appear as one large computing resource
MPI
A library of subprograms that can be called from C or FORTRAN

to write parallel program running on distributed computer systems
MapReduce
A web programming model for scalable data processing on large

clusters over large data sets, or in web search operations
Hadoop
A software library to write and run large user applications on vast

sets in business applications (http://hadoop .apache.org/core)
35
Programing on Cloud & Distributed Computing

Parallel and Distributed Programming Models & Tool Sets.
Model
Description
Features
PVM
PVM is a software package that

permits a heterogeneous collection
of serial, parallel and vector
computers which are connected to a
network to appear as one large
computing resource
Parallel Virtual Machine (PVM) uses the message

passing model to allow programmers to exploit
distributed computing across a wide variety of
computer types, including MPPs..PVM transparently
handles all message routing, data conversion, and
task scheduling across a network of incompatible
computer architectures.
MPI
A library of subprograms that can be

called from C or FORTRAN to write
parallel program running on
distributed computer systems
Specify synchronous or asynchronous point-to-point

and collective communication commands and I/O
operations in user programs for message-passing
execution
MapReduce
A web programming model for

scalable data processing on large
clusters over large data sets, or in
web search operations
Map function generates a set of intermediate

key/value pairs; Reduce function merges all
intermediate values with same key
Hadoop
A software library to write and run

large user applications on vast sets
in business applications
(http://hadoop .apache.org/core)
A scalable, economical, efficient, and reliable tool for

providing users with easy access of commercial
clusters
36
Cloud Prog. & Software Environments

Motivation :Simplicity of writing parallel programs in an
important metric for parallel and distributed programming
paradigms. Motivations behind parallel and distributed
programming models are :
1. To improve productivity of programmers
2. To decrease programs time to market
3. To leverage underlying resources more efficiently
4. To increase system throughput, and
5. To Support higher levels of abstraction
37

MapReduce : The software framework abstracts the data
flow of running a parallel program on a distributed
computing system by providing users with two interfaces
in the form of two interfaces : Map and Reduce
Users can override these two functions to interact with
and manipulate the data flow of running their program.
In MapReduce framework, the value part of the data
(key, value), is the actual data, and the key part is only
used by the MapReduce controller to control the data flow.
38

Definition of MapReduce : The MapReduce software

framework provides an abstraction layer with the data flow
and flow of control to users, and hide the implementation
of all data flow steps such as
data partitioning
mapping
Synchronization
Communication
Scheduling
39

Recently promoted proposed Parallel and distributed

Programming Paradigms
1. MapReduce
2. Hadoop
3. Dryad
Used for Information Retrieval Applications

Traditional Parallel Computing Models : MPI
40
Programing on Cloud and

Programming Enviornment
41
A long history of research efforts in message passing
P4
Chameleon
Parmacs
TCGMSG
CHIMP
NX (Intel i860, Paragon)
PVM (Got PVM Now ! -Good for Distributed Computing)
MPI (Got MPI now !. - Good for Large Multi-Processing)
Evaluate the needs of your application then choose (PVM or MPI)

MPI is a standard, has a steeper learning curve and doesnt have a standard way
to start tasks. MPICH does have an mpirun command
If building a new scalable, production code, should use MPI (widely supported
now)
Source : MPI References
42
Message Passing Architecture (MPI) Model

Message-Passing Programming Paradigm : Processors are connected
using a message passing interconnection network.
COMMUNICATION
NETWORK
P
M
On most Parallel Systems, the processes involved in the execution of a

parallel program are identified by a sequence of non-negative integers.
If there are p processes executing a program, they will have ranks 0,
1,2, ., p-1.
43
The Minimal Set of MPI Routines

The MPI library contains over 125 routines
But fully functional message-passing programs can be written
using only the following 6 MPI routines
MPI_Init
Initializes MPI
MPI_Finalize
Terminates MPI
MPI_Comm_size
Determines the number of processes
MPI_Comm_rank
Determines the label of the calling process
MPI_Send
Sends a message
MPI_Recv
Receives a message
All 6 functions return MPI_SUCCESS upon successful completion,

otherwise return an implementation-defined error code
All MPI routines, data-types and constants are prefixed by MPI_

All of them are defined in mpi.h ( for C/C++)
44
Starting and Terminating the MPI Library

1.#include <mpi.h>
2.main(int argc, char *argv[ ] )
3. {
4. MPI_Init(&argc, &argv );
5. // do some work
6. MPI_Finalize( );
7. }
Both MPI_Init and MPI_Finalize must be called by all
processes
Command line should be processed only after MPI_Init

No MPI function may be called after MPI_Finalize
45
An Overview Message Passing : Parallel Virtual Machine (PVM)
Parallel Virtual Machine (PVM) uses the message

passing model to allow programmers to exploit
distributed computing across a wide variety of computer
types, including MPPs.
When a programmer wishes to exploit a collection of
networked computers, they may have to contend with
several different types of heterogeneity:
architecture
data format
computational speed
machine load and
network load.
Source : PVM References
46
An Overview Message Passing : Parallel Virtual Machine (PVM)
PVM in Nutshell
Each host (could be an MPP or SMP) runs a PVMD
A collection of PVMDs define a virtual machine
Once configured, tasks can be started (spawned), killed,
signaled from a console
Basic message passing
Performance is OK, but API Semantics limit optimizations
PVM Strengths : interoperability; fault tolerance;
heterogeneity; resource control; dynamic model
Source : PVM References
47

MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Partitioning function
Map worker
Partitioning
function
Partitioning
function
Partitioning
function
Partitioning
function
Regions
Reduce worker
Use of MapReduce partitioning function to link the Map and Reduce workers.
48

MapReduce, Twister, and Iterative MapReduce
User interfaces
Input
Files
Map
Reduce
Function Flow of data Function
Output
Files
Abstraction
layer
Control flow
MapReduce
software
framework
MapReduce Library
Controller
MapReduce framework. Input data flows through the Map and Reduce functions
to generate the output result under the control flow using MapReduce software
library. Special user interfaces are used to access the Map and Reduce resources.
49
Cloud Prog. & Software Environments : Parallel and Distributed Programming Paradigms
Map worker
Input split
Map worker
Input split
Map worker
Input split
K3:v k5:v
K1:v k2:v k1:v k3:v K1:v k2:v k3:v k2:v
K3:v
K3:v k5:v
K1:v
Map function
M
K4:v k4:v K3:v
k4:v
Sort and group
Sort and group
Sort and group
K1:v k2:v k3:v
K1:v k2:v k3:v
K3:v k4:v
R1
R2
k3:v
k5:v k1:v
R1
R2
k4:v k3:v
R1
R2
k2:v k3:v
Data flow implementation

of many functions in the
Map workers and in the
Reduce workers through
multiple sequences of
partitioning, combining,
synchronization and
communication, sorting
and grouping, and reduce
operations.
k1:v
Input file
partitioning
Combiner
Partitioning
Synchronization
communication
R1
R2
k2:v k3:v k3:v k4:v k3:v
Sort and group

k2:v k3:v k3:v k4:v k3:v
R
k1:v k5:v k1:v
Sort and group
Sorting and grouping
k1:v, v, v k5:v
R
Output file
Output file
Reduce worker
Reduce worker
Reduce
Output
50

User program
(2) Fork
Controller
(2) Fork
(2) Fork
User program
Master
(3) Assign
map task
(3) Assign
reduce task
Worker
User program
Split 2
(4) Read
(5) Read
User program
(9) Communications
Worker
(12) Write
(11) Reduce
Worker
User program
Split 3
File 1
Output
files
User program
Worker
File 1
Split 1
Input
files
User program
Worker
Control flow implementation of the MapReduce functionalities in Map workers and Reduce workers
(running user programs) from input files to the output files under the control of the master user program.
51

Summary of the following steps :
How to describe the map workers. ?
Reading the Input data (data distribution)

Each Map worker reads its corresponding portion of the
input data, namely the input data split and sends it to its
Map function.
Although a map worker may run more than one Map
function, which means it has been assigned more than one
input data split, each worker is usually assigned one inout
split only
52

MapReduce Logical Dataflow
The input data to the Map function is in the form of a
(key, value) pair.
The output data from the Map function is structured as
key, value) pairs called intermediate (key value) pairs.
The user-defined Map function processes each input
(key, value) pair and produces a number of (zeros, one
or more) intermediate (key, value) pairs.
The goal is to process all input (key, value) pairs to the
Map functions in parallel.
53

MapReduce Logical Dataflow
The Reduce function receives the intermediate (key, value)
pairs in the form of a group of intermediate values associated
with one intermediate key (key, [set of values] ).
The MapReduce framework forms these groups by first
sorting the intermediate (key, value) pairs and then grouping
values with the same key.
Data is sorted to simplify the grouping process.

The Reduce function processes each (key, [set of
values] ) group and produces a set of (key, value) pairs as
output.
54

The Main Responsibility of the MapReduce Framework : Run a users
program on a distributed computing system. Therefore, the MapReduce
framework meticulously handles all
data partitioning
mapping
Synchronization
Communication and
Scheduling
Details of such data flows are required to address overheads and
performance issues from scalability point of view.
55

1. Data Partitioning : The MapReduce library splits the input
data (files) already stored in GFS into M pieces that also
correspond to the number of map tasks.
56

2. Computation Partitioning : This is implicitly handled (in
the MapReduce framework) by obliging users to write
their programs in the form of the Map and Reduce
functions. Therefore, the MapReduce library only
generates copies of a user program (e.g., by a fork system
call) containing the Map and Reduce functions, distributes
them, and starts them up on a number of available
computation engines.
57

3. Determining the master and workers : The MapReduce
architecture is based on a master-worker model.
Therefore, one of the copies of the user program
becomes the master and the rest become workers. The
master picks idle workers, and assigns the Map and
Reduce tasks to them.
58

3. Determining the master and workers : The MapReduce
architecture is based on a master-worker model.
What is Compute Engine ?
A map/reduce worker is typically a computation engine such
as a cluster node to run map/reduce tasks by executing
Map/Reduce functions.
59
Programing on Cloud and

Application Challenges
60
Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms
Where are Opportunities ?

Application-oriented Opportunities Through
Artificial Intelligence Algorithms
High Performance Comp. for massive graphs
Streaming
Visualization of large Data Sets
Heterogeneous Systems under CLOUD Comp.
Combined use of Many-core (PARAM YUVA-II) (A
combination of Co-processors & GPU Accelerators)
Power-aware Computing Energy Efficient
61
Graphs are pervasive in large-scale data analysis

Graphs are spared through-out in large-scale data analysis
Source of massive data : Petascale simulations, internet, scientific applications

Challenges : Data Size, heterogeneity, uncertainty, data quality.: Discover
emergent communities, model spread of Information
Astro-Physics
Problem : Outlier Detection of
Observations
Challenges: Massive Data Sets
Variation - timewise
Graph Problems: Data Mining
Alg. Association & Clusters;
Data Storage handling
Bio-Informatics
Problem : Drug Design
/Protein/Seq. Analysis
Challenges: Massive Data
Sets Data Heterogeneity
Graph Problems: Data
Mining Alg. Clustering; Fast
Query Searching Alg.
Bio-Informatics
Problem : Discover emergent
communities, Real-time Information
spread & Knowledge recovery
/Challenges: Massive Data Sets Data
Heterogeneity New analytics
Graph Problems: Data Mining Alg.
Clustering Shortest Path, Small /Large
Queries; Data Movement Irregular
Access
Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
62
Social Networks Analysis & Pattern Recognition

Application: Data Mining Large Scale Weather Simulation
Ocean and Land Temperature (Jan 1982)
Incremental Improvements :
Few Cores to Many Cores
Research Goals:
NPP
Pressure
Precipitation
Find global climate patterns of

interest to Earth Scientists
Parallel Data Mining
Global snapshots of values for a

number of variables on land surfaces
or water.
NPP
Pressure
Precipitation
SST
SST
Latitude
grid cell
Longitude
Data Mining
Time
zone
A key interest is finding

connections between the
ocean and the land. Large
Computing Power
Source - Reference : Vipin Kumar; Discovery of Patterns in the Global Climate System using Data Mining -
63
Many Core- Killer Apps of Tomorrow Parallel Alg. Design

Computer
Vision
Face
Body
Tracking Detection
(Financial)
Analytics
Physical
Simulation
Rendering
Global
Illumination
CFD
Face,
Cloth
Rigid
Body
Portfolio
Mgmt
Data Mining
Option
Pricing
Machine Cluster/ Text

learning Classify Index
Media
Synthesis
SVM
Classification
SVM
Training
PDE
Collision
detection
Level Set
Particle
Filtering
Filter/
transform
Fast Marching
Method
LCP
NLP
FIMI
IPM
(LP, QP)
K-Means
Monte Carlo
Index
Bench
Source : Intel
Krylov Iterative
Solvers (PCG)
Direct Solver
(Cholesky)
Basic matrix primitives

(dense/sparse, structured/unstructured)
Basic Iterative Solver

(Jacobi, GS, SOR)
Non-Convex
Method
Basic geometry primitives

(partitioning structures, primitive tests)
64


Opportunity 1 : Use of Massive Graph
Computations in High Performance Computing
Interactions: Vertices (No.s); Edges (Type of
Interactions); Time Wise variation; Non re-use
Non-Locality of Data : Distributed cache based systems
Massive Threads based models may be required
Data Movement Storage Energy Efficiency
A Focused research is required to design and implement new
class of algorithms based on Graph Computations
65

Opportunity 1 : Use of Massive Graph

Computations in High Performance Computing
O(Billion) vertices, O(Trillion) edges,
1 Million updates/sec
Challenge
Maintain Analytics, update quickly
Connected Components
Agglomerative Clustering
Problem :
Irregular execution
Bandwidth and latency bound
Irregular memory access
Low-to-no reuse
66


Opportunity 2 : Use of Graph Data Bases
Efficient Data Structures Few Billion vertices of
Graph Face Book- contains nodes and relationships;
Nodes contain properties (key-value pairs);
Relationships are named, directed and always have a
start and end node; Relationships can also contain
properties
Data Movement Storage Energy Efficiency
A Focused research is required to implement Graph based
Computations (FOCUS on GRAPH DATABASES)
67
Modeling Data with Graphs

Going Beyond Social Networks
Data Instance
Graphs are suitable for
capturing arbitrary relations
between the various
elements.
Graph Instance
Element
Elements Attributes
Relation Between
Two Elements
Vertex
Vertex Label
Edge
Type Of Relation
Edge Label
Relation between
a Set of Elements
Hyper Edge
Provide enormous flexibility for modeling the underlying data as they allow
the modeler to decide on what the elements should be and the type of
relations to be modeled
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
68

Graph Classification
Approach
Graphs with multi-dimensional labels

Stream graphs
phone-network connections
Hypergraphs
compact representation of set relations
Benchmarks and real-life test cases!
Graph
Databases
Discover Frequent
Sub-graphs
Transform Graphs
in Feature
Representation
Select Discriminating
Features
Learn a Classification
Model
Source : References & Nishith-Pathak, Analyzing Information Flow in Social Networks: Communities, Topics, Cognition and Influence, Doctor
of Philosophy Thesis, Department of Computer Science, University of Minnesota, Minneapolis, March 2012
69

Lake Superior: Coarse Mesh
Lake Superior: Mesh

Partitioning (by METIS) Lake Superior: Refined Mesh
Source : Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~kumar
70

Approaches for obtaining Coarsening graphs
The process of finding a maximal matching and contracting the
graph to obtain the next level coarser graph is important.
Coarsening graphs are obtained using maximal matchings Heavy edge

matching works very well. Significantly reduces the edge-weight left in the
graph. It requires very little refinement effort.
71

HyperGraph Computations
Multi-level graph Partitioning Algorithms
(b) Groups Determined by Ede-Coarsening
Various Ways of matching the vertices in the hypergraph connecting they induce
72


Opportunity 3: Quick Visualization Data
New tech. for Massive Graph Data
Scientific Computing Billion of Vertices with multiple
weights proportional to Floating Point Data
Graph Analytics for Discrete Data Million to few
Billion Vertices
Use od Data Mining & Artificial Intelligence Alg.
Context-based or Situation-aware based Computing Coprocessors & Accelerators
73

Opportunity 4: Combination of using

Heterogeneous Cloud Computing framework with
Many Core Systems
Solving Data analysis
Combination of Grid Computing & Cloud Computing
Combination of Hadoop & Parallel Programming for
Massively Multi-threading for SNA & BIG DATA
New Algorithms for Storage APIs for Data Science
74


Opportunity 5: Energy Efficiency Computing
Power aware Computing Methodologies (Calculate Power in
Mill-watts and Performance of Application Kernels)
Consumption of Power-in-Milliwatts : Understand Floating
Point or Memory Intensive Computing
Consumption of Power in Milliwatts: Movement of Data
Across different Levels of Storage
New Algorithms for less power consumption for SNA and
BIG-DATA
75

To work on each Opportunity

What is Required ?
Graph Algorithms - Large Scale Data Bases Graph - Data Mining Association & Clustering
Use Graph Data Bases (Irregular Computations
/Irregular Memory Access)
BIG Data Analytics Scientific Comp./ NonScientific Comp.) Data Parallelism Approach
76

To work on each Opportunity

What is Required ?
Storage Data Access in real time & Data
Management - Unstructured /Irregular Data
Storage HPC Storage (HDF5, Berkeley DB,
PLFS, netCDF, MPI-IO, SWIFT)
Hadoop implementation of MapReduce
77
Cloud and Distributed System Technologies

Conclusions
An Overview of Distributed Computing and Cloud
Computing
Programming on Cloud and Distributed Computing
Systems
Opportunities for Applications on Disturbed Computing
78
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by Kai
Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
Amazon EC2 and S3, Elastic Compute Cloud (EC2) and Simple Scalable Strorage (S3),
http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud and
http://spatten_presentations.s3.amazonaws.com/s3-pm-rails.pdf
F. Berman, G. Fox, T. Hey (Eds.), Grid Computing, Wiley, 2003.
K. Birman, Reliable Distributed Systems: Technologies, Web Services, and Applications, SpringerVerlag, 2005.
G. Boss, et al., Cloud Computing-The BlueCloud Project.
www.ibm.com/developerworks/webspehere/zones/hipods/, October 2007.
J. Dongarra, et al. (Eds.), Source Book of Parallel Computing, Morgan Kaufmann, San Francisco, 2003.
V. K. Garg, Elements of Distributed Computing, Wiley-IEEE Press, 2002.
K. Hwang, Advanced Computer Architecture: Parallelsim, Scalability, Programming, McGraw-Hill,
1993.
K. Hwang, Z. Xu, Scalable Parallel Computing, McGraw-Hill, 1998.
NVIDIA Corp. Kepler: NVIDIAs Next-Generation CUDA Compute Architecture, White paper, 2013.
I. Taylor, From P2P to Web Sertives and grids, Springer-Verlag, London 2005.
Twister, Open Source Software for Iterative MapReduce, http://www.iterativemapreduce.org/.
Wikipedia, Internet of Things, http://en.wikipedia.org/wiki/Internet_of_Things, June 2010.
Wikipedia, CUDA, http://en.wikipedia.org/wiki/CUDA, March 2011.
Wikipedia. TOP500, http://en.wikipedia.org/wiki/TOP500, February 2011.
79
References
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
D. Bader, R. Pennington, Cluster computing applications, Int. J. High Perform. Comput. (May) (2001).
R. Buyya (Ed.), High-Performance Cluster Computing. Vols. 1 and 2, Prentice Hall, New Jersey, 1999.
J. Dongarra, Survey of present and future supercomputer architectures and their interconnects, in:
International Supercomputer Conference, Heidelberg, Germany 2004.
Wikipedia, CUDA. http://en.wikipedia.org/wiki/CUDA, 2011, (accessed 19.02.2011).
R. Buyya, J. Broberg, A. Goscinski (Eds.), Cloud Computing; Principles and Paradigms, Wiley Press,
New York, 2011.
K. Hwang, D. Li, Trusted cloud computing with secure resources and data coloring, IEEE Internet
Comput., (September/October) (2010) 30-39.
D. Meyer, et. al., Parallax: Virtual disks for virtual machines, in: Proceedings of EuroSys, 2008.
L. Shri, H. Chen, J. Sun, vCUDA: GPU accelerated high performance computing in virtual machines, in:
Proceedings of the IEEE International Symposium on Parallel and Distributed Processing, 2009.
J. Smith, R. Nair, The architecture of virtual machines, IEEE Comput., (May) (2005).
J. Smith, R. Nair, Virtual Machines: Versatile Platforms for Systems and Processes, Morgan Kaufmann,
2006.
R. Ublig, et al. Intel virtualization technology, IEEE Comput., (May) (2005).
VMware (white paper). Understanding Full Virtualization, Paravirtualization, and Hardware Assist,
www.vmware.com/files/pdf/VMware_paravirtualization.pdf.
J. Walters, et al., A Comparison of virtualization technologies for HPC, in: Proceedings of Advanced
Information Networking and Applications (AINA), 2008.
K. Aberer, Z. Depotovic, Managing trust in a peer-to-peer information system, in: ACM CIKM
International Conference on Information and Knowledge Management, 2001.
Amazon EC2 and S3, Elastic Compute Cloud (EC2) and Simple Scalable Storage (S3).
http://spatten_presentations.s3amazonaws.com/s3-on-rails.pdf
80
References
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
M. Armbrust, A. Fox, R. Griffifth, et al., Above the Clouds: A Berkeley View of Cloud Computing, Technical
Report No. UCB/EECS-2009-28, University of California at Berkeley, 10 February 2009
I. Arutyun, et al., Open circus: a globalcloud computing testbed, IEEE Comput. Mag. (2010) 35-43.
G. Boss, P. Mllladi, et al., Cloud Computing: the bluecloud project.www.ibn.com/developerworks/
websphere/zones/hipods/, 2007.
R. Buyya, CS Yeo, S. Venugopal, Market-oriented cloud computing: vision, hype, and reality for delivering IT
services as a computing utilities, in: Proceedings of the 10th IEEE International Conference on High
Performance Computing and Communications (HPCC), Dalian, China, 25-27 September 2008.
I. Foster, The grid: computing without bounds, Sci. Am. 288 (4) (2003) 78-85.
V. Jinesh, Cloud Architectures, White paper, Amazon. http://aws.amazon.com/about-aws/whatsnew/2008/07/16/cloud-architectures-white-paper/.
W. Norman, M. Paton, T. de Aragao, et al., Optimizing utility in cloud computing through autonomous
workload execution, in: Bulletin of the IEEE Computer Society Technical Committee on Data Engineering,
2009.
S. Roschke, F. Cheng, C. Meinel, Intrusion detection in the cloud, in: IEEE International Conference on
Dependable. Autonomic, and Secure Computing (DASC 09), 13 December 2009.
J. Rittinghouse, J. Ransome, Cloud Computing: Implementation, Management, and Security, CRC Publishers
2010.
Salesforce.com, http://en/wikipedia.org/wikiSalesforce.com/,2010.
VMware, Inc., Migrating Virtual Machines with Zero Downtime, www.vmware.com/, 2010 (accessed 07).
Wikipedia Cloud computing, http://en.wikipedia.org/wiki/Cloud_computing, 2010 (accessed 26.01.10)
Wikipedia, Data center, http://en.wikipedia.org/wiki/Data_center, 2010 (accessed 26.01.10)
81
References
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
M. Hadley, Web Application Description Language (WADL), W3C Member Submission,

www.w3.org/Submission/wadl/, 2009 (accessed 18.10.10).
I. Foster, H. Kishimoto, A. Savva, et al., The open grid services architecture version 1.5, Open Grid Forum,
GFD.80, www.ogf.org/documents/GFD.80.pdf, 2006.
G. Fox, D. Gannon, A Survey of the Role and Use of Web Services and Service Oriented Architectures in
Scientific/Technical Grids, http://grids.ucs.indiana.edu/ptliupages/publications/ReviewofServices and
Workflow-IU-Aug2006B.pdf, 2006 (accessed 16.10.10)
Apache ActiveMQ open source messaging system, http://activemq-apache.org/.
NaradaBrokering open source content distribution infrastructure, ww.naradabrokering.org/.
Amazon Simple Queue Service (Amazon SQS), http://aws.amazon.com/sqs/.
Microsoft Azure Queues, http://msdn.microsoft.com/en-us/windowsazure/ff635854.aspx.
N. Wilins-Diehr, Special issue: Science gateways common community interfaces to grid resources Concurr.
Comput. Pract, Exper. 19 (6) (2007) 743-749.
D. Thain, T. Tannenbaum, M. Livny, Distributed computing in practice: the condor experience, Concurr, Pract.
Exper. 17 (2-4) (2005) 323-356.
Open Grid Computing Environments web site, www.collab-pgce.org (accessed 18.10.10).
E. Deelman, D. Gannon, M. Shields, I. Taylor, Workflows and e-Science: an overview of workflow system
features and capabilities, Future Generation Comp. Syst. 25 (5) (2009) 528-540, doi: http://dx.doi.org/
10.1016/j.future.2008.06.012
R. Allen, Workflow: An Introduction, Workflow Handbook. Workflow Management Coalition, 2001.
Kepler Open Xource Scientific Workflow System, http://kepler-project.org.
H. Gadgil, G. Fox, S. Pallickara, M. Pierce, Managing grid messaging middleware, in: Challenges of Large
Applications in Distributed Engironments (CLADE), 2006, pp. 83-91.
Condor home page, www.cs.wisc.edu/condor/.
82
References
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
Microsoft, Project Trident: A Scientific Workflow Workbench, http//research.microsoft.com/enus/collaboration/tools/trident.aspx, 2010.

Zend PHP Company, The Simple Cloud API for Storage, Queues and Table, http://www.simplecloug.org/
home, 2010.
NOSQL Movement, Wikipedia list of resources, http://en.wikipedia.org/wiki/NosQL, 2010.
NoSQL Link Archive, LIST OF NOSQL DATABASES, http://nosql-database.org/, 2010.
Amazon, Welcome to Amazon SimpleDB, http://docs.amazonwebservices.com/AmazonSimpleDB/latest/
DevelperGuide/index.html, 2010.
Apache, The CouchDB document-oriented database project, http://couchdb.apache.org/, 2010.
Pig! Platform for analyzing large data sets, http://hadoop.apache.org/pig/, 2010.
A. Grama, G Karypis, V. Kumar, A. Gupta, Introduction to Parallel Computing, second ed., Addison Wesley,
2003.
S. Hariri, M. Parashar, Tools and Environments for Parallel and Distributed Computing, Series on Parallel and
Distributed Computing, Wiley, 2004, ISBN: 978-0471332886.
L. Silva, Rajkumar. Buyya, Parallel Programming Models and paradigms, (2007).
G. Fox, MPI and MapReduce, in: Clusters, Clouds, and Grids for Scientific Computing CCGSC, Flat Rock, NC,
http://grids.ucs.indiana.edu/ptliupages/presentations/CCGSC-Sept8-2010.pptx, 8 September 2010.
J. Ekanayake, X. Qiu T. Gunarathne, S. Beason, G. Fox, High Performance Parallel Computing with Clouds and
Cloud Technologies, Cloud Computing and Software Services: Theory and Technologies, CRC Press (Taylor
and Francis), 2010.
Wikipedia, MapReduce, http://en.wikipedia.org/wiki/MapReduce, 2010 (accessed 06.11.10).
R. Lammel, Googles MapReduce programming model Revisited, Sci. Comput. Prog. 68 (3) (2007) 208-237.
S. Ghemawat, H. Gobioff, S. Leung, The Google File System, in: 19th ACM Symposium on Operating Systems
Principles, 2003, pp.20-43.
Google, Introduction to Parallel Programming and MapReduce, http://code.ggogle.com/edu/parallel/
mapreduce-tutorial,html.2010.
83
References
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
SALSA Group, Iterative MapReduce, http://www,iterativemapreduce.org/, 2010.

T. White, Hadoop: The Definitive Guide, Second ed., Yahoo Press, 2010.
Apache, HDFS Overview, http://hadoop.apache.org/hdfs/, 2010.
Apache! Pig! (part of Hadoop), http://pig.apache.org/hdfs/, 2010.
G. C. Fox, A. Ho, E, Chan, W. Wang, Measured characteristics of distributed and cloud computing
infrastructure for message-based collaboration applications, in: Proceedings of the 2009 International
Symposium on Collaborative Technologies and Systems, IEEE Computer Society, 2009, pp 465-467.
Eucalyptus LLC, White Papers. http://www.eucalyptus.com/whitepapers.
Nimbus, Cloud computing for science, http://www.nimbusproject.org, 2010.
Nimbus, Frequently Asked Questions, http://www.nimbusproject.org/docs/current/faq.html, 2010.
Amazon, Amazon Elastic Compute Cloud (Amazon EC2). http://aws.amazon.com/ec2.
Sector/Sphere, High Performance Distributed File System and Parallel Data Processing Engine.
http://sector.sourceforge.net
Open Stack, Open Source, Open Standards Cloud, http://openstack.org/index.php, 2010.
SALSA Group, Catalog of Cloud Material, http://salsaphc.indiana.edu/content/cloud-materials, 2010.
Microsoft Research, Cloud Futures Workshop, http://research.microsoft.com/en-us/events/
cloudfutures2010/default.aspx, 2010.
P. Chaganti, Cloud computing with Amazon Web Services, Part 1: Introduction When its smarter to rent
than to buy, http://www.ibm.com/developerworks/architecture/library/ar-cloudaes1/, 2008.
Cloud computing with Amazon Web Services, Part 2: Storage in the cloud with Amazon Simple Storage
Service (S3) Reliable, flexible, and inexpensive storage and retrieval of your data, http://www.ibm.com/
developerworks/architecture/library/ar-cloudaws2/, 2008.
P. Changanti, Cloud computing with Amazon Web Services, Part 3: Servers on demand with EC2, http://
www.ibm.com/developerworks/architecture/library/ar-cloudaws3/, 2008.
G. Fox, S. Bae, J. Ekanayake, X. Qiu, H. Yuan, Parallel Data Mining from Multicore to Cloudy Grids, book
chapter of High Speed and Large Scale Scientific Computing, IOS Press, Amsterdam, 2009
http://grids.ucs.indiana.edu/ptilupages/publications.CetratoWriteupJune11-09. pdf.
84
References
91.
W. Allcock, J. Bresnahan, R. Kettimuthu, et al., The globus stripped gridFTP framework and server, in:
Proceedings of the ACM/IEEE Conference on Supercomputing, 2005.
92. R. Aydt, D. Gunter, W. Smith, et al., A Grid Monioring Architecture, Global Grid Forum Performance
Working Group, 2002.
93. Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley &
Sons, 2009.
94. Dongarra, I. Fister, G. Fox, et al., Sourcebook of Parallel Computing, Morgan Kaufman Publishers,
2002.
95. L. Ferreira, et al., Grid Computing in Research and Education,
(http://www.redbooks.ibm.com/abstracts/sg246895.html?Open )
96. L. Ferreira, et al., Grid Computing in Research and Education,
(http://www.redbooks.ibm.com/abstracts/sg246649.html?Open )
97. I. Foster, C. Kesselman, S. Tuecke, The anatomy of the grid: enabling scalable virtual organizations, Int.
J High. Perform. Comput. Appl. 15 (3) (2001) 200.
98. I. Foster, Globus toolkit version 4: software for service-oriented systems, J. Comput. Sci. Technol, 21
(4) (2006) 513-520.
99. H. Jin. Challenges of grid computing, Advances in Web-Age Information Management. Lecture Notes
in Computer Science, 3739 (2005) 25-31.
100. I Taylor, From P2P to Web Services and Grids, Springer-Verlag, London, 2005.
101. D. Thain, T. Tannenbaum, M. Livny, Distributed computing in practice: the condor experience,
Concurrency. Comput. Pract. Exp., (2005) 323-356.
102. Wikipedia, Grid Computing. http://een.wikipedia.org/wiki/Grid_computing, (accessed 26.04.11)
85
References
103. S. Androutsellis, D. Spoinellis, A survey of P2P content distribution technologies. ACM,. Comput. Surv.
(December) (2004).
104. J. Buford, H. Yu, E. Lua, P2P Networking and Applications, Morgan Kayfmann, December 2008. Also
www. P2pna.com.
105. X. Cheng, J. Liu, NetTube: exploring social networks for peer-to-peer short video sharing, in:
Proceedings of IEEE Inforcom, March 2009.
106. P. B. Godfrey, S. Shenker, I. Stoica, Minimizing churn in distributed systems. in Proceedings of ACM
SIGCOMM, 2006.
107. K. Ross, D. Rubenstein, Peer-to-peer systems in: IEEE Inforcom, Hong Kong, 2004, (Tutorial slides).
108. M. Ambrust, A. Fox R. Giffith, et al., Above the Clouds: A Berkeley View of Cloud Computing, Technical
Report No. UCB/CCES-2009-28, University of California at Berkeley, 10 February 2009.
109. J. Ekanayke, T. Gunarathne, J. Qiu, Cloud technologies for bioinformatics applications, in: IEEE
Transactions on Parallel and Distributed Systems, accepted to appear, http://grids.ucs.indiana.edu
/ptliupages/publications/BioCloud_TPDS_Journal_Jan4_2010.pdf, 2011)
110. S. Garfinkel, Commodity grid computing with Amazons S3 and EC2, Login 32 (1) (2007) 7-13
111. Grid 5000 and ALLADIN-G5K: An infrastructure distributed in 9 sites around France, for research in
large-scale parallel and distributed systems. http://www.grid5000.fr/mediawiki/index.php/Grid5000:
Home, (accessed 20.11.10).
112. R. Grossman, Y. Gu, M. Sabala, et al., The open cloud testbed: Supporting open source cloud
computing systems based on large scale high performance, in: A. Doulamis, et al., (Eds.),
DynamicNetwork Services, Springer, Berlin Heidelberg, 2010, pp. 89-97.
113. J. Rittinghouse, J. Ransome, Cloud Computing: Implementation, Management and Security, CRC
Publisher, 2010.
86
References
114. H. Song, Exploring Facebook and Twitter Technologies for P2P Social Networking, in: EE 657 Final Project
Report, University of Sothern California, May 2010.
115. E. Walker, Benchmarking Amazon EC2 for high-performance scientific computing, Login 33 (5) (2008) page
18-23.
116. M. Weng, A Multimedia Social Networking Community for Mobile Devices, Tisch School of The Arts, New
York University, 2007.
117. M. Yigitbasi, A. losup, D. Epema, C-Meter: a framework for performance analysis of computing clouds, in:
International Workshop on Cloud Computing, May 2009.
118. PVM Parallel Virtual Machine A Users Guide and Tutorial For Networked Parallel Computing Al Geist,
Adam Beguelin, Jack Dongarra, Weicheng Jiang, Robert Mancheck, Vaidy Sunderam, The MIT Press
Cambrdge, Massachusetts, London England, 1994
119. pvmhome http://www.csm.ornl.gov/pvm/pvm_home.htmloo
120. David Browning, Embarrassingly Parallel Benchmark under PVM, Computer Science Corp., NASA Ames
Research Labs.
121. J.Dongara et al PVM- experiences, current status and Future Directions, ORNL- Tenessee.
122. Building Linux Clusters O-Reilly
123. A Performance Comparison of DSM, PVM, and MPI, Paul Werstein, Mark Pethick, Zhiyi Huang, University
of Otago, Dunedin, New Zealand , 2003
87
References
124. Andrews, Grogory R. (2000), Foundations of Multithreaded, Parallel, and Distributed
Programming, Boston, MA : Addison-Wesley
125. Butenhof, David R (1997), Programming with POSIX Threads , Boston, MA : Addison Wesley
Professional
126. Culler, David E., Jaswinder Pal Singh (1999),
Parallel Computer Architecture - A
Hardware/Software Approach , San Francsico, CA : Morgan Kaufmann
127. Grama Ananth, Anshul Gupts, George Karypis and Vipin Kumar (2003), Introduction to Parallel
computing, Boston, MA : Addison-Wesley
128. Intel Corporation, (2003), Intel Hyper-Threading Technology, Technical User's Guide, Santa Clara
CA : Intel Corporation Available at : http://www.intel.com
129. Shameem Akhter, Jason Roberts (April 2006), Multi-Core Programming - Increasing Performance
through Software Multi-threading , Intel PRESS, Intel Corporation,
130. Bradford Nichols, Dick Buttlar and Jacqueline Proulx Farrell (1996), Pthread Programming O'Reilly
and Associates, Newton, MA 02164,
131. James Reinders, Intel Threading Building Blocks (2007) , OREILLY series
132. Laurence T Yang & Minyi Guo (Editors), (2006) High Performance Computing - Paradigm and
Infrastructure Wiley Series on Parallel and Distributed computing, Albert Y. Zomaya, Series Editor
133. Intel Threading Methodology ; Principles and Practices Version 2.0 copy right (March 2003), Intel
Corporation
134. William Gropp, Ewing Lusk, Rajeev Thakur (1999), Using MPI-2, Advanced Features of the
Message-Passing Interface, The MIT Press..
135. Pacheco S. Peter, (1992), Parallel Programming with MPI, , University of Sanfrancisco, Morgan
Kaufman Publishers, Inc., Sanfrancisco, California
136. Kai Hwang, Zhiwei Xu, (1998),
Scalable Parallel Computing (Technology Architecture
Programming), McGraw Hill New York.
137. Michael J. Quinn (2004), Parallel Programming in C with MPI and OpenMP McGraw-Hill
International Editions, Computer Science Series, McGraw-Hill, Inc. Newyork
138. Andrews, Grogory R. (2000),
Foundations of Multithreaded, Parallel, and Distributed
Progrmaming, Boston, MA : Addison-Wesley
88
References
139. SunSoft. Solaris multithreaded programming guide. SunSoft Press, Mountainview, CA, (1996),
Zomaya, editor. Parallel and Distributed Computing Handbook. McGraw-Hill,
140. Chandra, Rohit, Leonardo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, and Ramesh Menon,
(2001),Parallel Programming in OpenMP San Fracncisco Moraan Kaufmann
141. S.Kieriman, D.Shah, and B.Smaalders (1995), Programming with Threads, SunSoft Press,
Mountainview, CA. 1995
142. Mattson Tim, (2002), Nuts and Bolts of multi-threaded Programming Santa Clara, CA : Intel
Corporation, Available at : http://www.intel.com
143. I. Foster (1995, Designing and Building Parallel Programs ; Concepts and tools for Parallel
Software Engineering, Addison-Wesley (1995
144. J.Dongarra, I.S. Duff, D. Sorensen, and H.V.Vorst (1999), Numerical Linear Algebra for High
Performance Computers (Software, Environments, Tools) SIAM, 1999
145. OpenMP C and C++ Application Program Interface, Version 1.0". (1998), OpenMP Architecture
Review Board. October 1998
146. D. A. Lewine. Posix Programmer's Guide: (1991), Writing Portable Unix Programs with the Posix.
1 Standard. O'Reilly & Associates, 1991
147. Emery D. Berger, Kathryn S McKinley, Robert D Blumofe, Paul R.Wilson, Hoard : A Scalable
Memory Allocator for Multi-threaded Applications ; The Ninth International Conference on
Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX).
Cambridge, MA, November (2000). Web site URL : http://www.hoard.org/
148. Marc Snir, Steve Otto, Steyen Huss-Lederman, David Walker and Jack Dongarra, (1998) MPI-The
Complete Reference: Volume 1, The MPI Core, second edition [MCMPI-07].
149. William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William
Saphir, and Marc Snir (1998) MPI-The Complete Reference: Volume 2, The MPI-2 Extensions
150. A. Zomaya, editor. Parallel and Distributed Computing Handbook. McGraw-Hill, (1996)
151. OpenMP C and C++ Application Program Interface, Version 2.5 (May 2005), From the OpenMP
web site, URL : http://www.openmp.org/
152. Stokes, Jon 2002 Introduction to Multithreading, Super-threading and Hyper threading Ars
Technica, October (2002)
153. http://www.cdac.in/opecg2009/
89
References
154.Stokes, Jon 2002 Introduction to Multithreading, Super-threading and Hyper threading Ars Technica,
October (2002)
155.Andrews Gregory R. 2000, Foundations of Multi-threaded, Parallel and Distributed Programming,
Boston MA : Addison Wesley (2000)
156.Deborah T. Marr , Frank Binns, David L. Hill, Glenn Hinton, David A Koufaty, J . Alan Miller, Michael
Upton, Hyperthreading, Technology Architecture and Microarchitecture, Intel (2000-01)
157.http://www.erc.msstate.edu/mpi/
158.http://www.arc.unm.edu/workshop/mpi/mpi.html
159.http://www.mcs.anl.gov/mpi/mpich
160.The MPI home page, with links to specifications for MPI-1 and MPI-2 standards :
http://www.mpiforum.org
161.Hybrid Programming Working Group Proposals, Argonne National Laboratory, Chiacago (2007-2008)
162.TRAC Link : https://svn.mpi-forum.org/trac/mpi-form-web/wiki/MPI3Hybrid
163.Threads and MPI Software, Intel Software Products and Services 2008 - 2009
164.Sun MPI 3.0 Guide November 2007
165.Treating threads as MPI processes thru Registration/deregistration Intel Software Products and
Services 2008 - 2009
166.Intel MPI library 3.2 - http://www.hearne.com.au/products/Intelcluster/edition/mpi/663/
167. http://www.erc.msstate.edu/mpi/
168. http://www.arc.unm.edu/workshop/mpi/mpi.html
169. http://www.mcs.anl.gov/mpi/mpich
167. The MPI home page, with links to specifications for MPI-1 and MPI-2 standards :
http://www.mpiforum.org
168. Hybrid Programming Working Group Proposals, Argonne National Laboratory, Chiacago (2007-2008)
169. TRAC Link : https://svn.mpi-forum.org/trac/mpi-form-web/wiki/MPI3Hybrid
170. Threads and MPI Software, Intel Software Products and Services 2008 - 2009
171. Sun MPI 3.0 Guide November 2007
90
References
172. Treating threads as MPI processes thru Registration/deregistration Intel Software Products and
Services 2008 - 2009
173. Intel MPI library 3.2 - http://www.hearne.com.au/products/Intelcluster/edition/mpi/663/
174. http://www.cdac.in/opecg2009/
175. PGI Compilers http://www.pgi.com
176. David A. Bader, Massive-Scale Graph Analytics, Georgia Tech College of Computing, Department of
Computational Science and Engineering, HiPC-2012, Pune, India
177. D.A. Bader, G. Cong, and J. Feo, On the Architectural Requirements for Efficient Execution of Graph
Algorithms, The 34th International Conference on Parallel Processing (ICPP 2005), pp. 547-556,
Georg Sverdrups House, University of Oslo, Norway, June 14-17, 2005.
178. D.A. Bader and K. Madduri, Design and Implementation of the HPCS Graph Analysis Benchmark on
Symmetric Multiprocessors, The 12th International Conference on High Performance Computing
(HiPC 2005), D.A. Bader et al., (eds.), Springer-Verlag LNCS 3769, 465-476, Goa, India, December
2005.
179. Kamesh Madduri, David A. Bader, Jonathan W. Berry, Joseph R. Crobak, and Bruce A. Hendrickson,
Multithreaded Algorithms for Processing Massive Graphs, in D.A. Bader, editor, Petascale
Computing: Algorithms and Applications, Chapman & Hall / CRC Press, Chapter 12, 2007.
180. Cloud Applications Architectures Building Applications and Infrastructure in the Cloud George
Reese O-REILLY June 2011
181. Cloud Computing Bible Barrie Sosinsky, Wiley India Ltd, 2011
182. Cloud Computing Explained John Rhoton, British Library Cataloguing in Publication Data, 2009
183. Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by Kai
Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
180. Cloud Computing Infrastructure http://www.ibm.com
181. Cloud Computing Virtualization Made Easy http://www.cisco.com/
182. Cloud Computing Infrastructure http://www.softlyaer.com
183. Microsoft Cloud OS : http://www.microsoft.com
184. Amazon EC2 Cloud Discover Windows Azure Storage http://www.windowsazure.com
91
References
185.
186.
187.
188.
189.
190.
191.
192.
193.
194.
195.
196.
197.
198.
199.
200.
201.
202.
203.
204.
205.
Programming Windows Azure http://www.amazon.com

App Engine Google Cloud Platform https://cloud.google.com/appengine
Google App Engine Google Developers; https://developers.google.com/appengine/
Oracle NoSQL Database Architecture & Implementation http://www.oracle.com/NOSQLDatabase
NoSQL - MangoDB www.mongodb.com
Cloud Computing Meghdoot : http://www.cdac.in
High level overview of Object based (Swift) and block based storage http://www.openstack.org/software/openstack-storage/
Storage types in Eucalyptus : http://www.eucalyptus.com/
Graph Partitioning Software METICS /ParMETIS /HMETIS http://www.cs.umn.edu/~kumar
Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. 2009. Social influence analysis in large-scale networks. In Proceedings of the
15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 09). ACM, New York, NY, USA,
807-816
Nishith-Pathak, Analyzing Information Flow in Social Networks: Communities, Topics, Cognition and Influence, Doctor of
Philosophy Thesis, Department of Computer Science, University of Minnesota, Minneapolis, March 2012
Prof. Jaideep Srivastava ; http:/www.cs.umn.edu/~jaideep
G.Karypis, V. Kumar, and Vipin Kumar, Multilevel k-way partitioning scheme for irregular graphs, Jorunal of Parallel and
Distributed Computing, 48:96-129, 1998
The Anatomy of the Grid Enabling Scalable Virtual Organizations by Ian Foster, Carl Kesselman & etc..
Foster, I. Internet Computing and the Emerging Grid. Nature Web Matters, 2000.
http://www.nature.com/nature/webmatters/grid/grid.html
Foster, I. and Karonis, N. A Grid-Enabled MPI: Message Passing in Heterogeneous Distributed Computing Systems. In Proc.
SC'98, 1998.
The Globus CoG Home Page. http://www.globus.org/cog.
Globus. http://www.globus.org.
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov.
TeraGrid; http://www.teragrid.org/
92
Thank You
Any Questions ?
93

Cloud&Distributed Computing

Uploaded by

Copyright:

Available Formats

Cloud&Distributed Computing

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cloud&Distributed Computing

Uploaded by

Copyright:

Available Formats

An Overview of Cloud & Distributed Computing -

Pune University Campus

National Workshop on Big Data Analytics (BiDA2014) at CRRao AIMSCS

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

VCV. Rao, Ph.D

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed and Cloud Computing Prog. & Software Environments An Overview

Following topics will be discussed

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

References & Source for Presentation

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Centre for Development of Advanced Computing (C-DAC)

premier R&D organization of the Department of

Electronics and Information Technology (DeitY), Ministry of

Communications & Information Technology (MCIT) for

The setting up in 1988 - C-DAC has been undertaking

from PARAM with 1 GF in 1988.

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models & Enabling Technologies

Scalable Computing over the Internet

Computing Paradigm Distinctions

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models and Enabling Technologies

Scalable Computing over the Internet

Supercomputers (massively parallel processors or MPPs)

computers out of desire to share computing resources).

computer nodes that are physically connected in close

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models & Enabling Technologies

Scalable Computing over the Internet

Peer-to-peer (P2P) networks are formed for distributed file

Change will take place on HPC Paradigms to an HTC

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed Comp. & Software Environments

Source : Distributed and Cloud Computing Book, From Parallel Processing to

Emerging Parallel Processing Platforms: Challenges

Scalability Vs. Single System Image

Scalability Vs. Single System Image

Single System Image

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Shared Pool of Computing Applications

Guarantee at least one

Deliver large % of collective

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Emerging Parallel Processing Platforms: Challenges

Cluster of Multi-Core Systems

Multi Core Node 1

Cluster of Multi Core Processor Systems

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Emerging Parallel Processing Platforms: Clusters

top-level switches (288 ports)

node-level switches (24 ports)

Web Windows Other Subsystems

Single System Image Infrastructure

Gigabit Commodity Interconnect/SAN