Parallel Programming: Lecture #9

PARALLEL PROGRAMMING Lecture #9
1
Agenda
Principles of parallel Algorithm Design
o Introduction
• Decomposition, Tasks, & Dependency Graphs
• Granularity, concurrency & task interaction
• Processes & mapping
o Decomposition Techniques
1. Recursive Decomposition
2. Data Decomposition
3. Exploratory decomposition
2
4. Speculative decomposition
o Algorithm development is a critical component in solving a problem using computer
o Sequential Algorithm → a sequence of basic steps for solving a given problem using
serial computer
o Parallel Algorithm → a sequence of steps that tells us how to solve a problem

using multiprocessors and handle concurrency
o In practice, parallel algorithm my include some or all of the following:
✓ Identifying portions of the work that can be performed concurrently
✓ Mapping the concurrent pieces on to multiple processor
✓ Distributing input, output and intermediate data of the program
✓ Managing accesses to data shared by multiple processors
✓ Synchronizing the processors at various stages of program execution

3
o Two key steps in designing parallel algorithm
1. Dividing a computation into smaller computation.
2. Assigning them to different processors.
4
Decomposition, Tasks, &
Dependency Graphs
Decomposition
o The process of dividing a computation into tasks or smaller parts executed
in parallel
Tasks
o Are programmer defined units of computations
o Tasks may be of same, different, or even intermediate sizes.
Dependency Graphs
o Abstraction used to express dependencies among tasks (order of
execution). It is a directed graph with nodes corresponding to tasks and
edges indicating next task 5
Example: Multiplying a Dense Matrix with a Vector.
oComputation of each element of output vector y is independent of other

elements. Based on this, a dense matrix-vector product can be decomposed
into n tasks.
oObservations: While tasks share data (namely, the vector b ), they do not
have any control dependencies - i.e., no task needs to wait for the (partial)
completion of any other.
6
oAll tasks are of the same size in terms of number of operations.
Granularity, Concurrency & Tasks
interaction
Granularity
o The number of tasks into which a problem is decomposed determines
its granularity.
Fine grained Coarse-grained

Large number of small tasks Small number of large tasks
7
Fine grained
Coarse-grained
8
Example: Database Query Processing
Consider a relational database of vehicles, Each row is record contains data of a vehicle such ID,
model, year , color,…
Consider the computation to execute of the query:
MODEL = ``CIVIC'' AND YEAR = 2001 AND (COLOR = ``GREEN'' OR COLOR = ``WHITE)
9
o The previous query is processed by creating a number of intermediate
tables (ex 4 tables)
1. Table contain all civics

2. Table contain all 2001 model cars
3. Table contain all green cars
4. Table contains all white cars
o The computation proceeds by combining these tables by computing their pair

wise intersection or union
1. Compute intersection of table 1 & 2 → table 5 2001 model civic

2. Compute union of table 3 & 4 → table6
3. Compute intersection of table 5 & 6 → result
10
11
Note that the same problem can be decomposed into subtasks in other
ways as well.
12
Degree of Concurrency
o Max number of tasks that can be executed in parallel at a given time
o Maximum degree of concurrency <= total no of tasks …why?
Due to dependencies among tasks
o Since the number of tasks that can be executed in parallel may

change over program execution, the maximum degree of concurrency
is the maximum number of such tasks at any point during execution.
oWhat is the maximum degree of concurrency of the database query

examples?
13
4
14
Critical path:
oThe longest directed path between any pair of start node (node with
no incoming edge) and finish node (node with on outgoing edges).
• Critical path length:

oThe sum of weights of nodes along critical path.
15
Degree of Concurrency
o The average degree of concurrency is the average number of tasks
that can be processed in parallel over the execution of the program.
oAssuming that each tasks in the database example takes identical

processing time, what is the average degree of concurrency in each
decomposition?
oThe degree of concurrency increases as the decomposition becomes

finer in granularity and vice versa.
Average degree of concurrency
= total amount of work / critical path length
16
17
Critical path length = 27 Critical path length = 34
Average degree of concurrency Average degree of concurrency
= 63/27 = 2.33 = 64/34 = 1.88 18
Your Turn:
For this task graph, determine:
o degree of concurrency
o critical path
19
Task Interaction Graphs:
Are the dependency graph and interaction graph both the same?
oNote that task interaction graphs represent data dependencies,

whereas task dependency graphs represent control dependencies.
otask dependency graphs → output of one task is the input to other
oAlthough all tasks are independent they need access to specific data
oReturn to matrix example , they all need to access vector b , hence send
and receive messages to access the entire vector in distributed memory
20
Example:
oConsider the problem of multiplying a sparse matrix A with a vector
b. The following observations can be made:
oA matrix is said to be sparse if
1. Significant no of entries =zero
2. No of zero entries ≠predefined structure or pattern
Then this calculation can be optimized
21
Assume that task I is responsible for sending b[i] , hence task 4 is responsible
for sending b[4] to 0,5,8,9 and so on , then task interaction graph is shown in
the figure
22
Processes and Mapping
o Process → computing agent that perform tasks
o In general, the number of tasks in a decomposition exceeds the

number of processing elements available.
oFor this reason, a parallel algorithm must also provide a mapping of

tasks to processes.
oMappings are determined by both the task dependency and task

interaction graphs.
23
Objectives:
o Maximize concurrency: Task dependency graphs can be used
to ensure that work is equally spread across all processes at any
point (minimum idling and optimal load balance).
o Minimize Total Completion Time: map critical path tasks to the

same process.
o Minimize Communication: Task interaction graphs can be used

to make sure that processes need minimum interaction with other
processes.
24

Parallel Programming: Lecture #9

Uploaded by

Copyright:

Available Formats

Parallel Programming: Lecture #9

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Parallel Programming: Lecture #9

Uploaded by

Copyright:

Available Formats

PARALLEL PROGRAMMING Lecture #9

o Parallel Algorithm → a sequence of steps that tells us how to solve a problem

o In practice, parallel algorithm my include some or all of the following:

✓ Identifying portions of the work that can be performed concurrently

✓ Mapping the concurrent pieces on to multiple processor

✓ Distributing input, output and intermediate data of the program

✓ Managing accesses to data shared by multiple processors

✓ Synchronizing the processors at various stages of program execution

1. Dividing a computation into smaller computation.

2. Assigning them to different processors.

oComputation of each element of output vector y is independent of other

Fine grained Coarse-grained

1. Table contain all civics

o The computation proceeds by combining these tables by computing their pair

1. Compute intersection of table 1 & 2 → table 5 2001 model civic

o Since the number of tasks that can be executed in parallel may

oWhat is the maximum degree of concurrency of the database query

• Critical path length:

oAssuming that each tasks in the database example takes identical

oThe degree of concurrency increases as the decomposition becomes

oNote that task interaction graphs represent data dependencies,

otask dependency graphs → output of one task is the input to other

Then this calculation can be optimized

o In general, the number of tasks in a decomposition exceeds the

oFor this reason, a parallel algorithm must also provide a mapping of

oMappings are determined by both the task dependency and task

o Minimize Total Completion Time: map critical path tasks to the

o Minimize Communication: Task interaction graphs can be used

You might also like