University of Technology in Baghdad - Computer Engineering Department

Distributed Computer Systems

Issued by:
Dr. Ameer Mosa Thoeny Al-Sadi

Part1: Forms parallelism

Part2: Classification of Distributed Systems
(Flynn Taxonomy)

Reference:
ADVANCED COMPUTER ARCHITECTURE AND PARALLEL PROCESSING

Outlines

• Forms parallelism.
• Classification architecture:

A. Flynn Taxonomy: according to the dimensions of Instruction and Data.

1. SISD.
2. SIMD.
3. MISD.
4. MIMD.

B. According connection memory - processor:

1. Share memory:
-According access” UMA, NUMA, COMA”.
-According connection: buses, crossing switch, multistage network.

2. Message system:
-according network topology.

3. Hybrid system (combination access).

4. Distributed share memory.

Dr. Ameer Mosa Al-Sadi 1|P ag e

University of Technology in Baghdad - Computer Engineering Department

Form Parallelism
1. Processors:
-sequence: execute always only one instruction.
-”look ahead”: looking also to next instruction:
A technique for speeding up the process of fetching and decoding
instructions in a computer program. A central processing unit wherein
instruction fetch and execution is performed by a mechanism featuring
an instruction look ahead mechanism whereby fetching and processing
of the next software instruction is commenced as a last step of the
currently executing software instruction, and the currently executing
software instruction is terminated by the first portion of the next
software instruction.

2. Overlapping phase read and decode with executing instruction.

3. Function parallelism:
1) Multi-function units:
Multifunctional unit (MFU) of the processor. This reconfigurable
functional unit can hold several instructions or sequences of
instructions that can be provided by a special compiler, and loaded
by the processor when needed.

Dr. Ameer Mosa Al-Sadi 2|P ag e

University of Technology in Baghdad - Computer Engineering Department

2) Chaining-”pipeline”:
Pipelining imparts an implicit execution parallelism in the
different cycles of processing an instruction. Suppose execution of
an instruction consists of the following stages:
1. Fetch – Get the instruction from memory.
2. Decode – Determine what the instruction is.
3. Execute – Perform the instruction decode.
4. Write – Store the results to memory.

4. Chaining Program/Instructions (Next parts function units).

1) Implicit processing vector operand.
Exposed to Hardware.
Issue varying numbers of instructions per clock.

2) Explicit vector operand.

Parallelism is exposed to software (Compiler or Programmer).
Many different forms (Loosely coupled Multiprocessors to
tightly coupled VLIW).

Dr. Ameer Mosa Al-Sadi 3|P ag e

University of Technology in Baghdad - Computer Engineering Department

5. Register –register.
- Single Instruction Multiple Data “SIMD”:
ex: ”VLIW” (Very Long Instruction Word)
Processor array “EPIC” (Explicitly Parallel Instruction
Computing).

Very long instruction word (VLIW) describes a computer processing

architecture in which a language compiler or pre-processor breaks program
instruction down into basic operations that can be performed by the processor
in parallel (that is, at the same time). These operations are put into a very long
instruction word which the processor can then take apart without further
analysis, handing each operation to an appropriate functional unit.

“EPIC” ( Explicitly Parallel Instruction Computing ):

Its architecture should enable the CISC (complex instruction set computer)
processors to take a big enough step to overtake the RISC (Reduced
instruction set computer) processors. By using a technique called VLIW, still
experimental at the time, and creating the EPIC model, explicitly parallel.
That is, multiple instructions can easily be executed at once, assuming that the
hardware supports it.

6. Memory – memory.
- Multiple Instructions Multiple Data “MIMD”:
Ex: Multiprocessor, Multicomputer.

Dr. Ameer Mosa Al-Sadi 4|P ag e

University of Technology in Baghdad - Computer Engineering Department

Flynn Taxonomy
Michael J. Flynn (1966).
How much instruction is in given moment running and over
how much data elements?

Instruction Stream

Single Multiple

Single SISD MISD

Data

Stream
Multiple SIMD MIMD

FLYNN’S TAXONOMY OF COMPUTER ARCHITECTURE

The most popular taxonomy of computer architecture was defined

by Flynn in 1966.

Flynn’s classification scheme is based on the notion of a stream of

information. Two types of information flow into a processor: instructions
and data. The instruction stream is defined as the sequence of instructions
performed by the processing unit. The data stream is defined as the data
traffic exchanged between the memory and the processing unit.
According to Flynn’s classification, either of the instruction or data

Dr. Ameer Mosa Al-Sadi 5|P ag e

University of Technology in Baghdad - Computer Engineering Department

streams can be single or multiple. Computer architecture can be classified

into the following four distinct categories:

. single-instruction single-data streams (SISD);

. single-instruction multiple-data streams (SIMD);
. multiple-instruction single-data streams (MISD); and
. multiple-instruction multiple-data streams (MIMD).

Conventional single-processor von Neumann computers are classified as

SISD systems. Parallel computers are either SIMD or MIMD. When there
is only one control unit and all processors execute the same instruction in
a synchronized fashion, the parallel machine is classified as SIMD. In a
MIMD machine, each processor has its own control unit and can execute
different instructions on different data. In the MISD category, the same
stream of data flows through a linear array of processors executing
different instruction streams. In practice, there is no viable MISD
machine; however, some authors have considered pipelined machines
(and perhaps systolic-array computers) as examples for MISD.
An extension of Flynn’s taxonomy was introduced by D. J. Kuck in 1978.
In his classification, Kuck extended the instruction stream further to single
(scalar and array) and multiple (scalar and array) streams. The data stream
in Kuck’s classification is called the execution stream and is also extended
to include single (scalar and array) and multiple (scalar and array) streams.
The combination of these streams results in a total of 16 categories of
architectures.

Dr. Ameer Mosa Al-Sadi 6|P ag e

University of Technology in Baghdad - Computer Engineering Department

1) SISD ARCHITECTURE:

1. One control unit, one flow control.

2. Always read/write only one value data from memory.
3. Classical von Neumann architecture.

Input unit

control processor Memory

output unit

Today architecture of SISD:

CPU

North bridge
AGP RAM
(memory controller)

PCI System clock

south bridge
I/O
(I/O controller)

Dr. Ameer Mosa Al-Sadi 7|P ag e

University of Technology in Baghdad - Computer Engineering Department

2) SIMD ARCHITECTURE:

1. Single control unit (single instruction cache) control multi equal

(synchronize) executed units.
2. Equal operation will execute with various data.
3. Vector processors: operate over vector execute in single instruction
cycle.
4. (Increment over component scalar multiplies).
5. Work with special HW “VLIW, EPIC”.
Note: VLIW (Very Long Instruction Word),
EPIC (Explicitly Parallel Instruction Computing).
Scalar (in computing), an atomic quantity that can hold only one
value at a time.

The SIMD model of parallel computing consists of two parts: a front-end computer
of the usual von Neumann style and a processor array as shown in Figure, The
processor array is a set of identical synchronized processing elements capable of
simultaneously performing the same operation on different data. Each processor in
the array has a small amount of local memory where the distributed data resides
while it is being processed in parallel. The processor array is connected to the
memory bus of the front end so that the front end can randomly access the local
processor memories as if it were another memory. Thus, the front end can issue
special commands that cause parts of the memory to be operated on simultaneously
or cause data to move around in the memory. A program can be developed and
executed on the front end using a traditional serial programming language. The
application program is executed by the front end in the usual serial way, but issues
commands to the processor array to carry out SIMD operations in parallel.
The similarity between serial and data parallel programming is one of the strong
points of data parallelism. Synchronization is made irrelevant by the lock–step
synchronization of the processors. Processors either do nothing or exactly the same
operations at the same time. In SIMD architecture, parallelism is exploited by
applying simultaneous operations across large sets of data. This paradigm is most
useful for solving problems that have lots of data that need to be updated on a
wholesale basis. It is especially powerful in many regular numerical calculations.

Dr. Ameer Mosa Al-Sadi 8|P ag e

University of Technology in Baghdad - Computer Engineering Department

There are two main configurations that have been used in SIMD machines. In the
first scheme, each processor has its own local memory. Processors can communicate
with each other through the interconnection network. If the interconnection network
does not provide direct connection between a given pair of processors, then this pair
can exchange data via an intermediate processor.

Dr. Ameer Mosa Al-Sadi 9|P ag e

University of Technology in Baghdad - Computer Engineering Department

3) MISD ARCHITECTURE:

1. Multiple Various operation over equal data :

- Data pass at once to all execution units (different parts
single data flow).
- Data pass to first execution unit and his output is input to
next …” Micro- pipeline”.

2. Ex: Carnegie-Mellon: Multi mini processor,

”C.mmp” (16p,SIMD,MISD,MIMD)

3. “Data flow” architecture, systolic array.

Systolic arrays are often hard-wired for specific operations, such as "multiply and
accumulate”

4. Only for integrity, Practically unused.

Dr. Ameer Mosa Al-Sadi 10 | P a g e

University of Technology in Baghdad - Computer Engineering Department

4) MIMD ARCHITECTURE:

1. Multi flow “control and data” _ each processor can execute

different program on itself data.
2. Can work asynchronous (on different from MISD).
3. Processors synchronous for access to share (resources) data.
4. Computer with multi-processor (SM-MIMD).
“Shared Memory –MIMD”
5. Multi computer connected in network (DM-MIMD).
“Distributed-Memory-MIMD”

Dr. Ameer Mosa Al-Sadi 11 | P a g e

University of Technology in Baghdad - Computer Engineering Department

Multiple-instruction multiple-data streams (MIMD) parallel architectures are made

Of multiple processors and multiple memory modules connected together via some
interconnection network. They fall into two broad categories: shared memory or
Message passing. the general architecture of these two categories.
Processors exchange information through their central shared memory in shared
memory systems, and exchange information through their interconnection
network in message passing systems.
A shared memory system typically accomplishes interprocessor coordination
through a global memory shared by all processors. These are typically server
systems that communicate through a bus and cache memory controller. The bus/
cache architecture alleviates the need for expensive multi ported memories and
interface circuitry as well as the need to adopt a message-passing paradigm when
developing application software. Because access to shared memory is balanced,
these systems are also called SMP (symmetric multiprocessor) systems. Each
processor has equal opportunity to read/write to memory, including equal access
speed.
A message passing system (also referred to as distributed memory) typically
combines the local memory and processor at each node of the interconnection
network.
There is no global memory, so it is necessary to move data from one local memory
to another by means of message passing. This is typically done by a Send/Receive
pair of commands, which must be written into the application software by a
programmer.

Multiprocessors:
-Narrow band- communicates through share memory.
-Single (share) addresses space.
- Assigned data getting by searching program on
Share memory locations.
-Semaphore, mutex, barrier.
Multicomputer:
-Free band-communication message system.
-Each process have self memory and address space.
-Located assigned data by effected step (instruction) in program.

Dr. Ameer Mosa Al-Sadi 12 | P a g e