2016 IEEE International Parallel and Distributed Processing Symposium Workshops

Modules to teach parallel and distributed computing

using MPI for Python and Disco

José Ortiz-Ubarri Rafael Arce-Nazario Edusmildo Orozco

Computer Science Department Computer Science Department Computer Science Department
University of Puerto Rico - Rı́o Piedras University of Puerto Rico - Rı́o Piedras University of Puerto Rico - Rı́o Piedras
Email: jose.ortiz23@upr.edu Email: rafael.arce@upr.edu Email: edusmildo.orozco@upr.edu

Abstract—The ability to design effective solutions using par- computer clusters, and specially in the LittleFe [15] educa-
allel processing should be a required competency for every com- tional Cluster [14]. The educational modules can be used to
puting student. However, teaching parallel concepts is sometimes introduce students PDC concepts such as: the map reduce and
challenging and costly, specially at early stages of a computer master-slave models, load distribution, fault tolerance, scaling,
science degree. For such reasons we present a set of modules to
and security and privacy. The programming exercises use the
teach parallel computing paradigms using as examples problems
that are computationally intensive, but easy to understand and Python MPI for Python and Disco (map-reduce) libraries to
can be easily implemented using the Python parallelization deemphasize attention to low-level details and facilitate solu-
libraries MPI for Python and Disco. tions to more elaborate problems. Each module is constructed
around a real-world or research problem to showcase the
Keywords—parallel computing, mpi, mapreduce, master worker importance of parallel computing in practice and to motivate
the students. An extended abstract about the modules was
presented in [14]. Here we extend the work and share our
I. I NTRODUCTION experiences deploying them in our CS department.
Hands-on programming exercises and code demonstra- Our modules include:
tions/experiments are an essential educational resource for
teaching introductory parallel and distributed computing (PDC) • One module with instructions to install MPI for
concepts. The design of effective and applicable exercises can Python and Disco to the Little Fe cluster.
be challenging to the instructor, especially since many libraries • Modules to introduce MPI and MapReduce.
or frameworks for parallel programming require intricate sys-
tem setups and training students in low-level details to get even • Hands On modules with computational problems to
the most basic programs to work. In our teaching experience, be solved in parallel with MPI or MapReduce.
low-level details in parallel computation exercises lead, among
other things, to student (and instructor) frustration and deter For those interested in testing/using our modules, source
the students from focusing on the essential concepts [13]. code can be found at [1].

Most of the available resources to teach parallelism us- II. W HY P YTHON ?

ing Python concentrate in multiprocessing programming [16].
Python also has a nice multi-threading library that can be used Currently, the most popular introductory teaching language
to introduce PDC concepts. However it does not provide multi- at top US universities is Python [9]. In our computer science
core scalability [2]. The multiprocessing library also introduces department, C++ is the main language for the introduction
the concept of inter-process communication through message to programming and data structure courses, but most of the
passing. Nonetheless, using the multiprocessing library re- advanced courses use Python as their main language for
quires students to learn a new set of inter-process communica- assignments and projects.
tion constructs instead of using established standards such as
the message passing interface (MPI). Furthermore, nowadays The MPI for Python library provides bindings to the MPI
the Map reduce model is widely used in industry and academia standard and support for communications of general Python
to solve big data problems [8]. objects including communication of numeric arrays which
significantly ease the passing of data from one process to
We believe that the MPI for Python [6] library and the another [7]. For a quick comparison, Figure 6 has side by
Disco framework [12] are simple enough to introduce students side implementations of the classic MPI Hello World program
PDC concepts while giving them an experience close to what is in C++ and Python.
used in real-world high performance computing applications.
We also have found that bypassing the need to implement locks What the program does is very simple. When the program
in the modules is an advantage especially when the modules is executed num procs is the number of processes created that
are used in early CS core courses. will execute the same code. Each process gets an unique rank
from 0 to num procs - 1. If the process has the rank 0, it
We have designed a set of easy-to-deploy hands-on educa- displays the message and then receives the message from each
tional modules that can be deployed in multi-core computers, other process and displays it when it is received. All the other

978-1-5090-3682-0/16
/16 $31.00 © 2016 IEEE
$31.00 © 2016 IEEE 958
DOI 10.1109/IPDPSW.2016.204
Fig. 1. Hello world listings in Python and C++ MPI.

TABLE I. P OTENTIAL USE OF OUR MODULES IN CS COURSES AND

processes from rank 1 to rank num procs - 1 just send the THEIR B LOOM COGNITIVE LEVELS (K, C, A). K: K NOW THE TERM , C:
message to the process with rank 0. C OMPREHEND SO AS TO PARAPHRASE / ILLUSTRATE , A:A PPLY IT IN SOME
WAY.
From the code it is easy to see that for each function in
Python there is a matching C++ version. Notice that in the Module Topics CS0 CS1 CS2 Adv
Python version the data passing part is simplified by not having
to specify a buffer size in the send and receive function. In MPI Master / Worker Concurrency C C A A
for Python you could replace the string message by any general Costas arrays Tasks and Threads K C A A
Python object without further MPI specifications contrary to Performance metrics K C A A
the case of C++ where to send an object, you will need to know Divide and conquer K K C A
the memory layout and then build an special MPI data-type. Message Passing K K A A
Reduction K K A A
Disco is a lightweight, open-source framework for dis-
tributed computing based on the MapReduce paradigm. Disco Div & Conq Concurrency C C A A
is very powerful and easy, thanks to the Python platform. It password Tasks and Threads K C A A
is similar to the more popular Hadoop framework, and is also cracking Performance metrics K C A A
used for the distributed processing of large data sets across Divide and conquer K K C A
Message Passing K K A A
clusters of computers.
Reduction K K A A

III. T HE MODULES Map reduce Concurrency C C A A

network traffic Tasks and Threads K C A A
In this section we will present some of the modules de-
Performance metrics K C A A
veloped to introduce parallel computing in our CS curriculum Divide and conquer K K C A
and workshops for undergraduate students.The topics used in Message Passing K K A A
each of the modules have shown to be engaging for students. Reduction A A A A
Demonstrations of the power of parallel computing to solve
real-world problems could be done in any introductory CS
course. For instance the password cracking example of section
III-B was used to illustrate the power of parallel computing level (using Blooms [3] classification) that the student should
in our Introduction to Cybersecurity course, which is taken by achieve.
students with possibly no prior programming experience. The
other modules presented in this section where implemented in a
workshop. All the modules include serial and parallel solutions
to the problem. We decided to do so, in order to increase
the level of confidence of both instructors and students. Table As with any other educational material related to a real-
III shows the potential use of our modules in the courses of world application, the instructor should introduce the topic
a CS curriculum. Each course is matched with the learning with simple examples, appropriate notation, and adequate data

959
representations.

A. Master Slave Example using MPI for Python

In this section we will describe the parallel Master / Worker
implementation of the Costas arrays enumeration problem [4].
Costas arrays are known for their applications to communica-
tions and digital watermarking. The Costas array enumeration
problem consists of finding permutations, of a given size N ,
that meet the distinct difference property: for all integers h, i, Fig. 4. Computational branch of Costas arrays of size 6 that start with sub
and j, with 1 ≤ h ≤ n − 1, and 1 ≤ i, j ≤ n − h, permutation (1,3,2).
f (i + h)f (i) = f (j + h)f (j) implies i = j. Figures 2 and
3 show examples of permutations and analysis of Costas array
compliance.

Fig. 5. Master/worker diagram of Costas problem. Master sends sub-problems

to slaves. Slaves return the solution to the sub-problems.

sends work to the workers, then while there is still work to

do, it will retrieve results from the workers and sends more
Fig. 2. A Costas array, its matrix representation, and difference triangle,
which is used to check the distinct difference property. Observe that each row work to them. Finally when there is no more work to do, the
of the difference triangle does not contain repeated numbers. master collects the remaining results, and while it collects the
results, it notifies the workers to stop (TAG END). What to
do with the results will depend on the problem. For instance
the problem might require to store the results in a file, or that
some mathematical operation is performed to obtain a final
result.
The workers (rank > 0) will loop waiting for work from
the master, then perform the work and send back the results
until there is no more work to do.
The functions getW ork() and handleW ork() are what
essentially distinguish the master-worker implementation for
Fig. 3. A permutation that is not a Costas array. Observe that we can abort a particular problem. Function getW ork() should return the
checking the distinct difference property as soon as a difference is repeated input that will be sent to the workers. Function handleW ork()
in any h. should perform the work over the input and generate the result
that will be sent back to the master. In the case of the Costas
A naı̈ve approach to find all the Costas arrays of size N is problem illustrated in Figure 5, the getW ork() function should
to generate all N ! permutations. This search space can greatly return a sub-Costas array of size 3 (while there are arrays to
be reduced by using a backtracking algorithm that continues be processed), and the function handleW ork() must generate
searching a computational branch as long as it maintains the all the Costas arrays of size 6 (if any) that start with the sub-
Costas property or stop searching in the computational branch Costas array received.
otherwise (See Figure 4). However even this approach grows
factorially with N , thus the enumeration for sizes as small as In our experience the Costas array enumeration problem
14 may require significant execution time, e.g. in the order of can be completed during a class period of 90 minutes if
hours. the instructor provides the Master / Worker generic template
and the algorithm to generate permutations (which requires
We use the Costas array enumeration problem to introduce the longest programming effort). The same problem can be
the Master / Worker paradigm using MPI for Python. As illus- used as an assignment by having students come up with the
trated in Figure 5, the master is in charge of generating sub- permutation algorithms and code for parallel distribution from
permutations up to a certain length, then the sub-permutations scratch. Another related and fairly straightforward assignment
are distributed to be completed independently by the workers. is to generate sequences that solve the n–Queens problem [5].
When the workers/slaves complete the search for Costas arrays
of size N that start with the sub-permutation sent by the B. Divide and Conquer Example using MPI for Python
master, the results are sent back to the master.
Password cracking is a method by which an intruder that
The code in Figure 6 presents a simple and generic template gains access to an encrypted passwords file can reveal the
of the master / worker model. The master (rank = 0) initially original (clear text) passwords using a dictionary and brute

960
 Fig. 7. Map/Reduce diagram of the password cracking problem. Node
0 broadcasts the password file to all nodes. The dictionary is divided in
approximately the same size for each node. Cracked passwords are sent back
to the master.

classroom, and the topic has resulted to be very engaging to

the students. In very early introductory courses this module
has been used to show the students the scalability achieved
by dividing the problem in sub-problems and using parallel
computing. In more advanced courses the students are asked
to implement the parallel algorithm, after being introduced to
MPI.
The main struggle that our students exhibit when facing this
kind of problem is dividing an input of size M equally among
Fig. 6. Generic Python template for the Master / Worker model N nodes when M is not a multiple of N . They either fail
to recognize that they have left part of the problem unsolved,
e.g. distribute  M
N  jobs to every node, or choose to assign
force attack. Each word in a dictionary is encrypted and
compared to the encrypted passwords in the password file. the remaining jobs to one of the nodes, e.g. assign  M N +
In our opinion, the password cracking problem combines a (M mod N ) to the last node. The latter option is less harmful
set of desired properties in an exercise to illustrate parallel than the former. However the execution time of the program
computation. Firstly, it relates to cybersecurity - currently hot will be bounded by the node with the unbalanced load while
area of interest in our students. Secondly, it is easy to explain. the rest of the nodes may be idle for a significant part of the
Thus, even when a student does not fully understand the time. We ask the students to reflect on the problem, followed
parallel concepts, hes been shown how easy it is to recover by a discussion of the code in Listing 1.
weak passwords. Listing 1. Given a problem size, the number of processes and the rank of
the processes compute the portion of the problem for the rank
It has been found that the use of real-world problems to def computeIndices(size, nprocs, rank):
teach introduction to programming to undergraduates results portion = size / nprocs
in better retention and engagement of minorities in computing residue = size % nprocs
related fields [10], [11]. In our experience, we also have
observed that applications and programming problems related if rank <= residue:
to Cybersecurity are engaging and motivating for a wide group start = rank * (portion+1)
of students. else:
start = rank * portion + residue
We use the password cracking example to teach how to
solve problems by dividing the problem in sub-problems of finish = start + portion
approximately the same size among the available computing if rank < residue:
nodes. In our simple approach the password file is sent to all finish += 1
the compute nodes (MPI broadcast), and the dictionary file is
divided (approximately) equal to the compute nodes. Notice return start, finish
that we could have opted to broadcast both the password file
and the dictionary file. Our decision to broadcast only the C. Map/Reduce Example using Disco
password file promotes a discussion in the classroom about
file sizes and communication delays. The instructor can justify The aggregation of traffic data per host helps to identify
the implementation decision with several arguments: (1) the anomalies in a computer network like: (1) an IP in use that is
password file size is much smaller than the dictionary file, (2) not delegated, (2) an IP generating more traffic than normal,
the dictionary file is commonly installed with the operating (3) an IP that is not generating traffic.
systems of the compute nodes while the password file is not.
NetFlow is a network protocol that aggregates connection
We have deployed this module in workshops and in the information from one host to another over a certain period

961
of time (flow). NetFlow files consist of large quantities of libraries widely used in high performance computing applica-
connection information such as IP addresses of the source and tions. The modules use real-world examples to engage students
destination hosts, and the traffic in each connection (e.g., 5 in the hands-on experience. We also present PDC concepts that
minutes of UPR network traffic can be as high as 6.5MB, or can be introduced per course of the CS curriculum.
363,149 lines of flows).
The presented modules are easy to set up by the instructor
even in low-cost parallel platforms such as the LittleFe. Their
focus on real-world examples makes them a great resource
to provide hands-on experiences on PDC concepts to students
starting at the introductory courses. The modules make use of
libraries used in real high performance computing applications,
setting the stage for more advanced exercises in later courses
and/or industrial or academic research experiences for students.

ACKNOWLEDGMENT
The authors would like to thank the NSF-SFS award DUE-
Fig. 8. Map/Reduce diagram of the NetFlow data processing example. 1245744 for supporting this work.

In this module, we use NetFlow data processing as an R EFERENCES

example of an application that can be accelerated using Disco [1] Modules to teach pdc concepts - source code.
MapReduce. Simplified code for this application is presented https://bitbucket.org/cheojr/cpath/.
in Listings 2 and 3, where in (2) the source and destination IPs [2] threading - Higher-level threading interface, Python Standard Library.
are mapped by their addresses and valued by their traffic octets https://docs.python.org/2/library/threading.html. Accessed:2016-01-13.
(3) the reducer gathers the data, sorts the data by keyvalues, [3] Lorin W Anderson, David R Krathwohl, and Benjamin Samuel Bloom.
and computes their aggregated sums. The results generated by A taxonomy for learning, teaching, and assessing: A revision of Bloom’s
this simple code is a list of IP addresses with the amount of taxonomy of educational objectives. Allyn & Bacon, 2001.
data that has been transferred in the Network2. [4] Rafael Arce-Nazario, Jose R Ortiz-Ubarri, et al. Enumeration of Costas
arrays using GPUs and FPGAs. In Reconfigurable Computing and
Listing 2. Map function using Disco FPGAs (ReConFig), 2011 International Conference on, pages 462–467.
def map(line, params): IEEE, 2011.
#Split the data into an array [5] D Bollman and E Orozco. A Faster Algorithm for the n-Queens
data = line.split() Problem. Congressus Numerantium, pages 193–200, 2001.
yield data["sIPAddr"], int(data["Octets"]) [6] Lisandro Dalcı́n, Rodrigo Paz, and Mario Storti. MPI for Python.
yield data["dIPAddr"], int(data["Octets"]) Journal of Parallel and Distributed Computing, 65(9):1108 – 1115,
2005.
[7] Lisandro Dalcı́n, Rodrigo Paz, Mario Storti, and Jorge D’Elı́a. MPI for
Listing 3. Reduce function using Disco Python: Performance improvements and MPI-2 extensions. Journal of
def reduce(line, params): Parallel and Distributed Computing, 68(5):655 – 662, 2008.
for ip, traffic in kvgroup(sorted(iter)): [8] Jens Dittrich and Jorge-Arnulfo Quiané-Ruiz. Efficient big data pro-
yield ip, sum(traffic) cessing in Hadoop MapReduce. Proceedings of the VLDB Endowment,
5(12):2014–2015, 2012.
Listing 4. Main function using Disco to print the IP address in the network [9] Philip Guo. Python is now the most popular introductory teaching
with the amount of traffic they have generated. language at top us universities. BLOG@ CACM, July, 2014.
if __name__ == ’__main__’: [10] Lucas Layman, Laurie Williams, and Kelli Slaten. Note to self: make
job = Job().run(input=["file:///netflow.txt"], assignments meaningful. ACM SIGCSE Bulletin, 39(1):459–463, 2007.
map=map, reduce=reduce, params=None) [11] Jane Margolis and Allan Fisher. Unlocking the clubhouse: Women in
computing. MIT press, 2003.
for ip, traffic in result_iterator(job.wait(
[12] Prashanth Mundkur, Ville Tuulos, and Jared Flatow. Disco: A comput-
show=True)): ing platform for large-scale data analytics. In Proceedings of the 10th
print(ip, traffic) ACM SIGPLAN Workshop on Erlang, Erlang ’11, pages 84–89, New
York, NY, USA, 2011. ACM.
Disco provides such a high-level interface to the Map/Re- [13] Edusmildo Orozco, Rafael Arce-Nazario, Jose Ortiz-Ubarri, and Hum-
duce environment that the network flow example can even be berto Ortiz-Zuazaga. A Curricular Experience With Parallel Computa-
used in introductory programming courses. Its performance can tional Thinking: A Four Years Journey. In Proceeding of EduPDHPC,
2013.
be compared to a serial implementation using dictionaries in
Python. In upper-level courses the example is also useful to [14] José Ortiz-Ubarri and Rafael Arce-Nazario. Modules to teach parallel
computing using Python and the LittleFe cluster. Super Computing,
exemplify the trade-offs between master/worker vs. Map/Re- 2013.
duce, e.g. the improved speedup in MPI is only achieved after [15] Charles Peck. LittleFe: parallel and distributed computing education
the effort of hand-coding a good distribution of work to the on the move. Journal of Computing Sciences in Colleges, 26(1):16–22,
nodes. 2010.
[16] Sushil K Prasad, Anshul Gupta, Arnold L Rosenberg, Alan Sussman,
IV. C ONCLUSION and Charles C Weems. Topics in parallel and distributed computing:
Introducing concurrency in undergraduate courses. 2015.
In this work we present a set of modules that can be used to
introduce PDC concepts in the CS curriculum using standard

962