Introduction to Scientific

Programming

1
 Outline

• Overarching Goal: Computational Thinking (CT) + Parallel Computing Techniques

• Knowledge Base

• Programming Models

• C/C++ Language

• Specific Course Objectives

2
 3

Computational Thinking (CT)

• Jeannette Wing

computational thinking (CT) is a set of problem-solving methods that

involve expressing problems and their solutions in ways that a computer
could also execute.

https://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf

• https://ocw.mit.edu/courses/6-006-introduction-to-algorithms-fall-2011/

• https://www3.cs.stonybrook.edu/~pramod.ganapathi/CSE595.html
 Parallel Computing
• a process of breaking down larger problems into smaller,
independent, often similar parts that can be executed simultaneously
by multiple processing/computing elements communicating via some
communication mechanisms (e.g., shared memory, TCP/IP protocol);
the results of which are combined upon completion as part of an
overall algorithm.

• The primary goal of parallel computing is to increase available

computation power for faster application processing and problem
solving.

4
 Why Parallel Computing
• Solving an 𝑛𝑛 × 𝑛𝑛 linear system Ax=b by using
1 3
Gaussian elimination takes ≈ 𝑛𝑛 flops.
3
• On Core i7 975 @ 4.0 GHz, which is capable of about
60-70 Gigaflops (109flops)

𝑛𝑛 flops time
1000 3.3×108 0.006 seconds
1000000 3.3×1017 57.9 days

5
 Different Levels of Parallelism
1. Task/thread level parallelism (e.g., multicore + distributed memory/shared

memory).

2. Data level parallelism (e.g., Graphic processing units (GPUs))

3. Instruction level parallelism (e.g., pipelining)

• dividing incoming instructions into a series of sequential steps (the eponymous
"pipeline") performed by different processor units with different parts of instructions
processed in parallel.
• Overlaps execution of instructions from multiple iterations of a loop. Computer can
executes instructions from different iterations in the same pipeline, so that pipelines
are kept busy without stalls.

6
 Languages for Writing Parallel Code
• Common solutions for creating parallel code are OpenMP (open multi-
processing), MPI (message passing interface) and GPUs (graphics processing
unit). Solutions are limited to the C/C++ or Fortran programming languages.

• Why not Python?

– Python uses interpreter (e.g., ‘py’), and cannot match the speed of compiled languages
such as C/C++ or Fortran.
– C/C++ is strongly typed and is more manageable than Python which is dynamically
typed. This feature also enhances the performance of C++.
– C++ supports OOP; Python partially supports.
– C++ can be used for system programming including writing Operating systems.
– We can also use C++ to learn low-level programming as the language is closer to
hardware. With Python, such a feat is not possible.
7
 8

Python vs. C++

• https://intro2ml.pages.doc.ic.ac.uk/autumn2021/modules/lab-cpp/introduction

• Dynamically typed vs. strongly typed.

• Both support OOP. But ??? vs. generic programming, template (good for reusable codes) etc.

• Memory management. C/C++ access by names, pointers; references.

• Huge amount of libraries. vs. somewhat limited.

No best languages, just “right” languages for the job.

 9

Knowledge Base
1. Computer Architecture (von Neumann machine)
• Memory, CPU, Op., process/thread, file system, I/O.
• See appendix: S_4_Comp_Arch

2. Software engineering (basic)

• Data structure: array, list, stack, queue, etc.
• Basic constructs of language; Structured programming.
• Algorithms: merge sort, quicksort, adding/removing from a linked
list, etc.
 10

3. Software engineering (advanced)

• Algorithmic thinking/Computational thinking.

• Structured design: interfaces (functions, class), data

structure, basic logical structures, nested loops, recursive
function.
• OOP.

• Reusable codes; Efficiency; Error handling; Debugging.

4. Science & engineering training

• Numerical linear algebra; differential equation; optimization;

• Domain specific knowledge, e.g., CFD, mechanics, soft matter

physics, network analysis…
5. Programming Languages
• C/C++, Python, etc.

• Parallel platform + standards

Programming Models
1. Imperative (code directly controls execution flow and state change)
1. Procedural programming
2. Object-oriented programming

2. Declarative (code declares properties of the desired result, but not how
to compute it)
• Functional programming

https://en.wikipedia.org/wiki/Programming_paradigm

11
 Procedural Programming Model
• implement the behavior of a computer program as procedures (functions, subroutines)
that call each other. The resulting program is a series of steps that forms a hierarchy of calls
to its constituent procedures.

• Languages that support: Fortran, C

• Development practices (enhance quality and lower development and maintenance costs.)
• Modularity and scoping
Modularity is about organizing the procedures of a program into separate modules -- each of which has a
specific and understandable purpose. Minimizing the scope of variables and procedures can enhance
software quality by reducing the cognitive load of procedures and modules. A program lacking modularity
or wide scoping tends to have procedures that consume many variables that other procedures also
consume. The resulting code is relatively hard to understand and to maintain.
• Sharing/reuse
specify a well-defined interface and be self-contained -- in particular via the software library.
12
 Object-Oriented Programming (OOP)
• based on the concept of objects, which can contain data and code: data in
the form of fields (often known as attributes or properties), and code in
the form of procedures (often known as methods). Computer programs
are designed by making them out of objects that interact with one
another.
• Languages: C++, Java, Python,…

13
 Functional Programming
• programs are constructed by applying and composing functions. It is a
declarative programming paradigm in which function definitions are
trees of expressions that map values to other values, rather than a
sequence of imperative statements which update the running state of
the program.
• Languages: C++11, Java, Python, Fortan95, …
• lambda, map, closures, functools

14
Program Flow Chart
http://www.programiz.com/article/flowchart-programming

15
16
 17

Specific Course Objectives

1. What this course is NOT about: TensorFlow; PyTorth; Python libraries.
2. C/C++: Pointer; Dynamic memory; I/O; Class; STL*; OOP concepts; …
3. CUDA: heterogeneous computing
• CUDA C/C++; transfer data between memories; generating parallel tasks; compiler; run CUDA program…

4. MPI: homogeneous computing

• MPI library implemented by C; compiler; launch MPI job…

5. Parallel Algorithm Design

6. Course Projects
• Program design: complete interface (Gaussian elimination)

• OOP: complete C++ class; operator overloading; constructors … (Gaussian elimination)

• OOP: C++ class hierarchy; abstraction … (finite difference for Poisson equation)

• CUDA: (image processing)

• MPI: (parallelize finite difference for Poisson equation)

• Final project