Algorithms

Lecture : Algorithmic Thinking and Arithmetic Problems

Imdad ullah Khan

Contents
1 Parity check of an integer 1
1.1 Parity Check: Formulation for large input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Parity Check: A Different Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Algorithmic thinking and Terminology 2

3 Algorithms Design Questions 3

3.1 What is the problem? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2 Is the algorithm correct? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.3 How much time does it take? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.3.1 Runtime: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.4 Can we improve the algorithm? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Analysis of Algorithms 4
4.1 Running Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.2 Elementary operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.3 Runtime as a function of input size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.4 Best/Worst/Average Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.5 Growth of runtime as size of input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

5 Basic Integers Arithmetic problems 5

5.1 Addition of two long integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Correctness of Algorithm for adding two integers given as arrays . . . . . . . . . . . . . . 6
Runtime of Algorithm for adding two integers given as arrays . . . . . . . . . . . . . . . . 6
5.2 Multiplication of two long integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Runtime of grade-school multiplication algorithm . . . . . . . . . . . . . . . . . . . . . . . 7
5.2.1 A reformulation of the multiplication problem . . . . . . . . . . . . . . . . . . . . . . . . . 8

6 Computability, Algorithm Design, Complexity, and Cryptography 8

6.1 Computability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.1.1 Halting Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Checking if a program halts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.1.2 Interesting applications of Halting problems . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Fermat’s Last theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Goldbach Conjecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6.1.3 Collatz Conjecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6.2 Camp 1: Algorithm Designers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.3 Camp 2: Complexity Theorists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.4 Intersection of the Two Camps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.5 Cryptography: Building on Complexity Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6.6 The Factorization Problem and its Hardness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6.7 Coin Tossing over the Phone: A Cryptographic Protocol . . . . . . . . . . . . . . . . . . . . . . . 11
6.8 Security of the Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1
7 Egg Dropping Problem 12

8 Recursion and Exponentiation of an integer to a power 12

8.1 Exponentiation by iterative multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.2 Exponentiation by recursive multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.2.1 Runtime of recursive exponentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.3 Exponentiation by repeated squaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.3.1 Runtime of repeated squaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

9 Matrix and Vectors Arithmetic 15

9.1 Dot Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Runtime of dot-prod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9.2 Matrix-Vector Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.3 Matrix Multiplication via dot product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.4 Matrix-Matrix Multiplication via Matrix-Vector Product . . . . . . . . . . . . . . . . . . . . . . . 17

1 Parity check of an integer

We are going to discuss a very easy problem that should be trivial for everyone to solve. This will help us
develop a thinking style of a computer scientist and establish a lot of terminology.

Input: An integer A
Output: True if A is even, else False

Solution: Here is a simple algorithm to solve this problem.

Algorithm 1 Parity-Test-with-mod
if A mod 2 = 0 then
return true

The above solution is written in pseudocode (more on it later). This algorithm solves the problem, however
following are some issues with it.
ˆ It only works if A is given in an int
ˆ What if A doesn’t fit an int and A’s digits are given in an array?
ˆ What if A is given in binary/unary/. . . ?
Note that these issues are in addition to usual checks to see if it is a valid input, i.e. to check that A is not a
real number, a string, or an image, etc.

1.1 Parity Check: Formulation for large input

Suppose the integer is very large, (n-digits long for some n > the computer word length) so it does not fit any
variable. Say the input integer is given as an array A, where each element is a digit of A (least significant digit
at location 1 of the array). For example

7 6 5 4 3 2 1 0
A=
5 4 6 9 2 7 5 8

In this case, the following algorithm solves the problem.

2
Solution:
By definition of even integers, we need to check if A[0] mod 2 = 0. If so then output A is even.

Algorithm 2 Parity-Test-with-mod
Input: A - digits array
Output: true if A is even, otherwise false
if A[0] mod 2 = 0 then
return true
else
return false

1.2 Parity Check: A Different Formulation

What if the mod operator is not available in the programming language, then we can manually check if the last
digit is an even digit, i.e. check if A[0] ∈ {0, 2, 4, 6, 8}.

Algorithm 3 Parity-Test-with-no-mod
Input: A - digits array
Output: true if A is even, otherwise false
if A[0] = 0 then return true
else if A[0] = 2 then return true
else if A[0] = 4 then return true
else if A[0] = 6 then return true
else if A[0] = 8 then return true
else
return false

2 Algorithmic thinking and Terminology

We saw an algorithm for a very basic problem, next we discuss how a computer scientist ought to think
about algorithms and what question one needs to ask about algorithms. We will then discuss some basic
arithmetic problems and algorithms for them without elaborating explicitly about these questions, but they will
be answered. Every students should be well versed with these algorithms, we discuss them to develop relevant
vocabulary and terminology, such as elementary operations, runtime, correctness. We will also establish the
pseudocode notation that we will use throughout this course.

An algorithm is defined as a procedure designed to solve a computational problem. When designing an algorithm,
the designer needs to keep in mind some important features which are to be used when formulating an algorithm:
ˆ Input: This is the data that is passed to the algorithm, and may be in different forms. For instance,
it could be a simple number, or a list of numbers. Other examples include matrices, strings, graphs,
images, videos, and many more. The format of input is an important factor in designing algorithms, since
processing it can involve extensive computations.
ˆ Size of Input: The size of input is a major factor in determining the effectiveness of an algorithm.
Algorithms which may be efficient for small sizes of inputs may not do the same for large sizes.
ˆ Output: This is the final result that the algorithm outputs. It is imperative that the result is in a form
that can be interpreted in terms of the problem which is to be solved by the algorithm. For instance, it
makes no sense to output ‘Yes’ or ‘No’ to a problem which asked to find the maximum number out of an
array of numbers.

3
 ˆ Pseudocode: This is the language in which an algorithm is described. Note that when we say pseu-
docode, we mean to write down the steps of the algorithm using almost plain English in a manner that
is understandable to a general user. Our focus will be the solution to the problem, without going into
implementation details and technicalities of a programming language. Given our background, we will be
using structure conventions of C/C + +/Java. We will see many examples of this later on in the notes.

3 Algorithms Design Questions

With these features in mind, an algorithm designer seeks to answer the following questions regarding any
algorithm:

3.1 What is the problem?

ˆ What is input/output?, what is the ”format”?
ˆ What are the “boundary cases”, “easy cases”, “bruteforce solution”?
ˆ What are the available “tools”?
We saw in the parity test problem (when the input was assumed to be an integer that fits the word size), how
an ill-formulated problem (ambiguous requirement specification) could cause problem. Similarly, when we had
to consider whether or not mod is available in our toolbox.
As an algorithm designer, you should never jump to solution and must spend considerable amount time on
formulating the problem, rephrasing it over and over, going over some special cases.

Formulating the problem with precise notation and definitions often yield a good ideas towards a solution. e.g.
both the above algorithms just use definitions of even numbers
This is implementing the definition algorithm design paradigm, one can also call it a bruteforce solution, though
this term is often used for search problem.
One way that I find very useful in designing algorithms to not look for a smart solution right away. I instead
ask, what is the dumbest/obvious/laziest way to solve the problem? What are the easiest cases? and what are
the hardest cases? where the hardness come from when going from easy cases to hard cases?

3.2 Is the algorithm correct?

ˆ Does it solve the problem? Does it do what it is supposed to do?
ˆ Does it work in every case? Does it always “produce” the correct output? (also defined as the correctness
of the algorithm)
ˆ Does the algorithm halts on every input (is there a possibility that the algorithm has an infinite loop, in
which case technically it shouldn’t be called an algorithm though). Must consider “all legal inputs”
The correctness of the three algorithms we discussed for the parity-test problem follows from definition of
even/odd and/or mod, depending on how we formulate the problem

3.3 How much time does it take?

ˆ Actual clock-time depends on the actual input, architecture, compiler etc.
ˆ What is the size of input?
ˆ What are elementary operations? How many elementary operations of each type does it perform?

4
 ˆ How the number of operation depends on the “size” of input
ˆ The answer usually is a function mapping the size of input to number of operations, in the worst case.
ˆ We discuss it in more detail in the following section

3.3.1 Runtime:
of the algorithms we discuused so far. The first algorithm performs one mod operation over an integer and one
comparison (with 0). The second algorithm performs one mod operation over a single-digit integer (recall size
of the input) and one comparison (with 0) The third algorithm performs a certain number of comparisons of
single digit integers. The actual number of comparison depends on the input. If A is even and its last digit is 0,
then it takes only one comparison, if A[0] = 2, then we first compare A[0] with 0 and then with 2, hence there
are two comparisons. Similarly if A is odd or A[0] = 8, then it performs five comparisons.
This gives rise to the concept of best-case, worst-case analysis. While the best-case analysis gives us an optimistic
view of the efficiency of an algorithm, common sense dictates that we should take a pessimistic view of efficiency,
since this allows us to ensure that we can do no worse than worst-case, hence our focus will be on worst-case
analysis.
Note that the runtimes of these algorithms do not depend on the input A, as it should not, as discussed above we
consider the worst case only. But they do not even depend on the size of the input, we call this constant runtime,
that is it stays constant if we increase the size of input, say we double the number of digits the algorithm still
in the worst case performs 5 comparisons.

3.4 Can we improve the algorithm?

ˆ Can we tweak the given algorithm to save some operations?
ˆ Can we come up with another algorithm to solve the same problem?
ˆ Computer scientists should never be satisfied unless ......
ˆ May be we can’t do better than something called the lower bound on the problem.
Since all three algorithms perform only a constant number of operation and we are usually concerned how can
we improve the runtime as the input size grows, in this case we do not really worry about improving it.

4 Analysis of Algorithms
Analysis of algorithms is the theoretical study of performance and resource utilization of algorithms. We
typically consider the efficiency/performance of an algorithm in terms of time it takes to produce output. We
could also consider utilization of other resources such as memory, communication bandwidth etc. One could
also consider various other factors such as user-friendliness, maintainability, stability, modularity, and security
etc. In this course we will mainly be concerned with time efficiency of algorithm.

4.1 Running Time

This is the total time that the algorithm takes to provide the solution. The running time of an algorithm is a
major factor in deciding the most efficient algorithm to solve a problem. There are many ways we could measure
the running time, such as clock cycles taken to output, time taken (in seconds), or the number of lines executed
in the pseudo-code. But these measures suffer from the fact that they are not constant over time, as in the case
of clock cycles, which varies heavily across computing systems, or time taken(in seconds), which depends on
machine/hardware, operating systems, other concurrent programs, implementation language, and programming
style etc.

5
We need a consistent mechanism to measure running time of an algorithm. This runtime should be inde-
pendent of the platform of actual implementation of the algorithm (such as actual computer architecture,
operating system etc.) We need running time also to be independent to actual programing language used
for implementing the algorithm.
Over the last few decades, Moore’s Law has predicted the rise in computing power available in orders of
magnitude, so processing that might have been unfeasible 20 years ago is trivial with today’s computers. Hence,
a more stable measure is required, which is the number of elementary operations that are executed, based on
the size of input. We will define what constitutes an elementary operation below.

4.2 Elementary operations

These are the operations in the algorithm that are used in determining the running time of the algorithm.
These are defined as the operations which constitute the bulk of processing in the algorithm. For instance, an
algorithm that finds the maximum in a list of numbers by comparing each number with every other number in
the list, will designate the actual comparison between two numbers (is a < b ?) as the elementary operation.
Generally, an elementary operation is any operation that involves processing of a significant magnitude.

4.3 Runtime as a function of input size

We want to measure runtime (number of elementary operations) as a function of size of input. Note that this
does not depend on machine , operating system or language. Size of input is usually number of bits needed to
encode the input instance, can be length of an array, number of nodes in a graph etc. It is important to decide
which operations are counted as elementary, so keep this in mind while computing complexity of any algorithm.
There is an underlying assumption that all elementary operation takes a constant amount of time.

4.4 Best/Worst/Average Case

For a fixed input size there could be different runtime depending on the actual instance. As we saw in the parity
test of an integer. We are generally interested in the worst case behavior of an algorithm
Let T (I) be time, algorithm takes on an instance I. The best case running time is defined to be the minimum
value of T (I) over all instances of the same size n.
Best case runtime:
tbest (n) = minI:|I|=n {T (I)}

Worst case runtime:

tworst (n) = maxI:|I|=n {T (I)}

Average case:
tav (n) = AverageI:|I|=n {T (I)}

4.5 Growth of runtime as size of input

Apart from (generally) considering only the worst case runtime function of an algorithm. We more importantly,
are interesting in
1. Runtime of an algorithm on large input sizes
2. Knowing how the growth of runtime with increasing input sizes. For example we want to know how the
runtime changes when input size is doubled?

6
5 Basic Integers Arithmetic problems
Now that we have established the terminology and thinking styles, we look at some representative problems and
see whether we can satisfy the questions with regards to the algorithm employed.

5.1 Addition of two long integers

Input Two long integers, given as arrays A and B each of length n as above.
Output An array S of length n + 1 such that S = A + B.

This problem is very simple if A, B, and their sum are within word size of the given computer. But for larger n
as given in the problem specification, we perform the grade-school digit by digit addition, taking care of carry
etc.
6 5 4 3 2 1 0 1 1 1
A=
4 6 9 2 7 5 8 5 4 6 9 2 7 5 8
+
6 5 4 3 2 1 0 8 5 1 7 2 2 6 1
B=
5 1 7 2 2 6 1 1 3 9 8 6 5 0 1 9
In order to determine the tens digit and unit digit of
a 2-digit number, we can employ the mod operator and
the divide operator. To determine the units digit, we The algorithm, with mod operator.
simply mod by 10. As for the tens digit, we divide by
10, truncating the decimal. We need this to determine Algorithm 4 Adding two long integers
the carry as we do manually. Can there be a better way Input: A, B - n-digits arrays of integers A and B
to determine the carry in this case, think of what is the Output: S = A + B
(largest) value of a carry when we add two 1-digit integer. 1: c ← 0
We can determine if there should be a carry if a number 2: for i = 0 to n − 1 do
is greater than 9, in that case we only have to extract 3: S[i] ← (A[i] + B[i] + c) mod 10
the unit digit. We discussed in class that if the mod 4: c ← (A[i] + B[i] + c)/10
operator and type-casting is not available, then we can 5: S[n] ← c
still separate digits in an integer by using the definition
of positional number system.

Correctness of Algorithm for adding two integers given as arrays The correction of this algorithm
again follows from the definition of addition.

Runtime of Algorithm for adding two integers given as arrays The running time (number of single
digits) arithmetic operations performed by this algorithm is determined in details as follows, later on we will
not go into so much detail. We count how many times each step is executed.

Algorithm 5 Adding two long integers

Input: A, B - n-digits arrays of integers A and B
Output: S = A + B 
1: c ← 0 1 time
2: for i = 0 to n − 1 do 

3: S[i] ← (A[i] + B[i] + c) mod 10 n times
c ← (A[i] + B[i] + c)/10

4:

5: S[n] ← c 1 time

7
We do not count the memory operations, the algorithm clearly performs 4n 1-digit additions, n divisions and
n mod operations. If we consider all arithmetic operations to be of the same complexity, then the runtime of
this algorithm is 6n. You should be able to modify the algorithm to be able to add to integers that do not have
the same number of digits.
Can we improve this algorithm? Since any algorithm for adding n digits integers must perform some
arithmetic on every digit, we cannot really improve upon this algorithm.

5.2 Multiplication of two long integers

Input Two long integers, given as array A and B each of length n as above.
Output An array C length 2n + 1 such that C = A · B.

We apply the grade-school multiplication algorithms, multi- Algorithm 6 Multiplying two long integers
ply A with the first digit of B, then with the second digit Input: A, B - n-digits arrays of integers A and B
of B and so on, and adding all of these arrays. We use a Output: C = A ∗ B
n × n array Z (a 2-dimensional array or a matrix) to store 1: for i = 1 to n do
the intermediate arrays. Of course, now we know how to add 2: c←0
these arrays. 3: for j = 1 to n do
2 7 5 8 4: Z[i][j + i − 1] ← (A[j] ∗ B[i] + c) mod 10
×
9 6 3 2 5: c ← (A[j] ∗ B[i] + c)/10
5 5 1 6 6: Z[i][i + n] ← c
8 2 7 4
7: carry ← 0
1 6 5 4 8
8: for i = 1 to 2n do
2 4 8 2 2
9: sum ← carry
2 6 5 6 5 0 5 6
10: for j = 1 to n do
Here again we will use the technique to determine the carry
11: sum ← sum + Z[j][i]
and the unit digits etc. Can we be sure that when we multi-
ply two 1 digit integers, the result will only have at most 2 12: C[i] ← sum mod 10
digits. 13: carry ← sum/10
14: C[2n + 1] ← carry

Runtime of grade-school multiplication algorithm

ˆ The algorithm has two phases, in the first phase it computes the matrix, where we do all multiplications,
and in the second phase it adds all elements of the matrix column-wise.
ˆ In the first phases two for loops are nested each running for n iterations. You should know that by the
product rule the body of these two nested loops is executed n2 times. As in addition, the loop body has
6 arithmetic operations. So in total the first phase performs 6n2 operations.
ˆ In the second phase, the outer loop iterates for 2n iterations which for each value of i the inner loop iterates
n times (different value of j. While in the body of the nested loop there is one addition performed, so
in total 2n2 additions. Furthermore, the outer loop (outside the nested loop) performs one mod and one
divisions, so a total of 2n arithmetic.
ˆ The grand total number of arithmetic operations performed is 6n2 + 2n2 + 2n = 8n2 + 2n.
ˆ Question, when we double the size of input, (that is make n double of the previous), what happens to the
number of operations, how do they grow. Draw a table of the number of operations for n = 2, 4, 8, 16, 32.

8
5.2.1 A reformulation of the multiplication problem
In this section, we demonstrate how the multiplication problem can be
reformulated and how it helps us solve the problem much simply.
Algorithm 7 Multiplying two long
Think of the integers A and B in positional number system and apply
integers using Distributive law of
distributive and associative laws we get the following very simple algo-
multiplication over addition
rithm.
1: C ← 0
    2: for i = 1 to n do
A[0]∗100 +A[1]∗101 +A[2]∗102 +. . . × B[0]∗100 +B[1]∗101 +B[2]∗102 +. . . 3: for j = 1 to n do
4: C ← C +10i+j ×A[i]∗B[j]
The algorithm with this view of the problem is as follows
The correctness of this algorithm also follows from definition of multiplication and it performs n2 single-digit
multiplications and n2 shifting (multiplication with powers of 10).
Can we improve these algorithms? When we study divide and conquer paradigm of algorithm design we
will improve upon this algorithm.

6 Computability, Algorithm Design, Complexity, and Cryptography

6.1 Computability
Computability theory, a branch of mathematical logic, computer science, and the theory of computation studies
questions like what problems can be solved by a computer (are there uncomputable problems) and if so, are all
noncomputable problems at the same level or is there a hierarchy among them. There is also a focus on defining
what is a computer? Interested students are referred to check out the Church-Turing thesis or wait for your
theory of Automata course.

6.1.1 Halting Problem

The Halting Problem is one of the most fundamental problems in theoretical computer science. It asks whether
a given program halts (finishes execution) for every possible input. You must have seen computer programs
that operate on other computer programs that are input to it, e.g., compilers, interpreters, debugger, operating
system). You also must have seen computer programs that never halts, e.g., those with an infinite loop or a
bottomless recursive function.
The halting problem formalizes the question of whether a program stops or not.
ˆ Input: A program P (.cpp) and an input x.
ˆ Output: Output: Yes if P.cpp halts on every x , else No
Well, the clear way is to run P on x (call P (x)) if it stops working, we output Yes, else we output No. But
how long should we wait before declaring No. Please think about it, and keep thinking until it is clear.
The Halting Problem is noncomputable (the technical word is undecidable), meaning no algorithm can solve this
problem for all possible program-input pairs. It can be proved with a simple application of the diagnolization
argument, that we discussed in detail in our discrete mathematics course.
I am going to highlight, the significance of the Halting Problem stems using (made-up) connection to other
interesting problems (some of which we are very familiar with).

Checking if a program halts For some programs, checking if they halt is easy

9
 Algorithm Integer n Algorithm No input
Algorithm No input n←1
while n ̸= 0 do
return true while n ̸= 0 do
n←n−2
n←n+1
Clearly halts
Halts if n ≥ 0 and n is even
Never halts

6.1.2 Interesting applications of Halting problems

Fermat’s Last theorem
Theorem 1. Fermat’s Last Theorem (1637) For n ≥ 3, an + bn = cn has no solution where a, b, c are positive
integers

Algorithm negFLT(integer n) Suppose we had a solution halt for the halting problem.
Calling halt.cpp on negFLT we get the following.
f lag ← true
a←1 halt(negFLT(n)) = Yes ⇐⇒ Fermat’s last theorem is
while f lag = true do false
for b = 1 → a do So, halt would have resolved the Fermat’s last theorem,
for c = 2 → a + b do it took about 300 years for us to assert its correctness. Ok!
if an + bn = cn then
we know FLT is true, are there other applications of the
f lag ← false
would halt solution
a←a+1

Goldbach Conjecture
Cojecture 1. Goldbach Conjecture (1742)? Every even integer n > 2 is the sum of two primes.

halt(negGoldbach(n)) = Yes ⇐⇒ Goldbach conjec-

ture is false Algorithm negGoldbach( even integer n)
Thus, an algorithm for halt(·) would resolve the Goldbach f lag ← true
n←2
conjecture
while f lag = true do
Recall my promise if you solve the Goldbach conjecture
f lag ← false
you will get an A. I would have promised A+ in this course
n←n+2
for solving the Halting problem, but it cannot be solved. for p = 2 → n do
By the way, above we only saw that halt would resolve if isPrime(p) and isPrime(n − p) then
Goldbach conjecture, which says nothing about Goldbach f lag ← true
conjecture itself begin true or false. break

6.1.3 Collatz Conjecture

The Collatz Conjecture, (also known as wondrous numbers, Algorithm Collatz Program ( integer n)
3n + 1 conjecture, Syracuse problem, Ulam conjecture) is an while n ̸= 1 do
open problem in mathematics and can be expressed as an iter- if n is even then
ative algorithm: n ← n/2
Cojecture 2. Collatz Conjecture (1937) For every integer n else
the Collatz Program eventually reaches 1 (thus halts) n ← 3n + 1

Here are some runs of this program. n = 3 =⇒ 3, 10, 5, 16, 8, 4, 2, 1

10
n=4 =⇒ 4, 2, 1
n=5 =⇒ 5, 16, 8, 4, 2, 1
n=6 =⇒ 6, 3, 10, 5, 16, 8, 4, 2, 1
n=7 =⇒ 7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 15, 8, 4, 2, 1
n=8 =⇒ 8, 4, 2, 1
n=9 =⇒ 9, 28, 14, 7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 15, 8, 4, 2, 1

27, 82, 41, 124, 62, 31, 94, 47, 142, 71, 214, 107, 322, 161, 484, 242, 121, 364, 182, 91, 274, 137, 412, 206, 103, 310, 155, 466, 233, 700,
350, 175, 526, 263, 790, 395, 1186, 593, 1780, 890, 445, 1336, 668, 334, 167, 502, 251, 754, 377, 1132, 566, 283, 850, 425, 1276, 638, 319,
958, 479, 1438, 719, 2158, 1079, 3238, 1619, 4858, 2429, 7288, 3644, 1822, 911, 2734, 1367, 4102, 2051, 6154, 3077, 9232, 4616, 2308,
1154, 577, 1732, 866, 433, 1300, 650, 325, 976, 488, 244, 122, 61, 184, 92, 46, 23, 70, 35, 106, 53, 160, 80, 40, 20, 10, 5, 16, 8, 4, 2, 1

Any other number checked so far, it does actually halts. But we do not have a proof of this. A solution to the
Halting problem will readily resolve this conjecture Despite its simplicity, the Collatz Conjecture has baffled
mathematicians and computer scientists for decades.

For about a month everyone at Yale worked on it, with no result. A similar phenomenon
happened when I mentioned it at the University of Chicago. A joke was made that this
problem was part of a conspiracy to slow down mathematical research in the U.S.

Shizuo Kakutani, 1960

Mathematics is not yet ripe enough for such questions.

Paul Erdös, 1983

In computer science (in the realm of computable problems), we can roughly categorize researchers into two
main camps: Algorithm Designers and Complexity Theorists. Each of these camps focuses on different but
interconnected aspects of problem-solving with computers.

6.2 Camp 1: Algorithm Designers

Algorithm Designers focus on creating efficient algorithms to solve computational problems. Earlier, we discussed
the thinking of algorithm designers.

6.3 Camp 2: Complexity Theorists

Complexity Theorists study problems to determine whether there is any algorithm that can solve them efficiently
or, in some cases, whether there is any algorithm that can solve them at all. The key questions that complexity
theorists ask are as follows, we will discuss them in detail later in this course.
ˆ What is the complexity class of the problem? Can it be solved in polynomial time (P class), or is
it NP-complete, meaning it is as hard as the hardest problems in NP (nondeterministic polynomial time)?
ˆ Can the problem be reduced to another well-known problem? Is there a known transformation
between this problem and another problem of known complexity?
These theorists are less focused on practical solutions and more interested in the theoretical boundaries of what
is possible to compute efficiently or at all.

6.4 Intersection of the Two Camps

While these two camps seem to focus on different aspects of computation, their work is deeply interconnected.
Many times, algorithm designers rely on the results from complexity theory to determine whether an efficient

11
algorithm can even exist for a given problem. Similarly, complexity theorists often use insights from algorithmic
techniques to classify problems.

6.5 Cryptography: Building on Complexity Results

One of the most interesting areas where complexity theory is applied in practice is in cryptography. Cryptog-
raphy leverages the hardness of certain computational problems to build secure communication protocols. The
security of these protocols often depends on the assumption that some problems are so computationally difficult
that no efficient algorithm can solve them in a reasonable amount of time.

6.6 The Factorization Problem and its Hardness

The factorization problem is one such example of a problem whose difficulty is leveraged in cryptography.
Given a large integer x that is the product of two prime numbers p and q, it is easy to multiply p and q to get
x, (we saw algorithms for them above and more in the slides).
Input: A long integer x (having n digits). x is product of two primes.
Output: Two prime numbers p, and q, such that x = p × q

I am giving briefly some ideas of solving the factorization problem. Suppose x has n digits.
1. Try all factors from 3 to x

This algorithm takes x steps, which (the value of x) could be O(10n )

√
2. Try all factors from 3 to x
√
x steps, which as above could be O(10n/2 )

3. Use number field sieve method

1/3
Its runtime is ∼ O(en ) This is essentially the best known method
The general assumption in the algorithms community that there is no efficient algorithms to factorize two
integers. Indeed, if this assumption is true, then one cannot tell anything meaningful about the factors p and q.
Let me state, the assumption, in the next section we will use it.
Assumption 1. There is no “efficient algorithm” to check any non-trivial property of p or q, for example,
checking if one of p and q have last digit 3.

6.7 Coin Tossing over the Phone: A Cryptographic Protocol

We can use the hardness of the factorization to design a protocol that allows
two parties to perform a fair coin toss over the phone, with no trusted Alice and Bob each want to
authority, without either party being able to cheat. win a coin flip over phone
ˆ Step 1: To call “head” Alice generates two large prime numbers, p
and q, such that none of them has last digit 3, to call “tails”, she
chooses two prime numbers so that at least one of them ends with a
3. She keeps p and q secret but sends N = p × q to Bob.
ˆ Step 2: Bob flips the coin, or some how choose either “heads” or
“tails”. Bob sends the outcome of the coin flip to Alice.
ˆ Step 3: Alice sends the values of p and q to Bob.

12
 ˆ Step 4: Bob verifies that p × q = N . Based on the values of p and q,
Bob checks whether Alice chose p and q according to the rules (neither
ends in 3 for heads, at least one ends in 3 for tails).

6.8 Security of the Protocol

The protocol’s security is based on the hardness of the factorization problem. Specifically:
ˆ Can Bob cheat? Bob cannot figure out p and q from N because factorizing N is computationally hard
(assumption). Hence, he cannot manipulate the result of the coin toss after Alice sends him N .
ˆ Can Alice cheat? Alice commits to N = p × q before Bob makes his call. After Bob makes his choice,
Alice cannot change N to produce a different pair of p and q that would unfairly alter the result, because
the factorization of N is fixed and verifiable.
– What if Alice choose p, q, r with only r ending with 3. Alice sends N = p × q × r, and reveals p and
qr or pq and r so she gets her choice? – well, in this case notice that qr and pq are not primes. And
primality testing can be done efficiently, hence Bob will catch this way of cheating.
In summary, this cryptographic protocol ensures fairness by relying on the computational difficulty of the
factorization problem. Neither party can cheat because of the complexity-theoretic hardness of reversing the
product of large prime numbers. This simple but powerful idea forms the foundation of many cryptographic
protocols, such as RSA encryption, and is a great example of how complexity results are used to solve real-world
problems.

7 Egg Dropping Problem

In the class, we also discussed the very interesting problem, The Egg Dropping Problem that seeks to determine
the critical/threshold floor in a building from which if an egg is dropped, it will break. The objective is to
minimize the number of drops required to find this critical floor in the worst-case scenario. Formally,
Input: A building with n floors and k eggs.
Output: The critical floor f , where the egg will break if dropped from floor f or higher.
There are various strategies to solve this problem, including:
ˆ Linear Search: Drop an egg starting from the first floor and go upwards. This approach guarantees that
we find the critical floor after at most n drops, but it is inefficient.
ˆ Binary Search: Start from the middle floor, and based on whether the egg breaks or not, move up or
down accordingly. This approach reduces the number of drops, but it requires unlimited eggs.
What can we do when there are two eggs or more.

8 Recursion and Exponentiation of an integer to a power

Input: Two integers a and n ≥ 0.
Output: An integer x such that x = an .

13
8.1 Exponentiation by iterative multiplication

We apply the grade-school repeated multiplication algorithm, i.e.

Algorithm 14 Exponentiation
just execute the definition of an and multiply a n times. More
Input: a, n - Integers a and n ≥ 0
precisely
n times Output: an
x = an = a ∗ a ∗ . . . ∗ a ∗ a
z }| { 1: x ← 1
2: for i = 1 to n do
3: x←x∗a
This way of looking at an leads us to the following algorithm.
4: return x

This algorithm clearly is correct and takes n multiplications. This time integer multiplications not 1-digit
multiplications. We can tweak it to save one multiplication by initializing x to a, but be careful what if n = 0.

8.2 Exponentiation by recursive multiplication

Exponentiation can also be performed by a recursive algorithm, in
case you recursively love recursion. We use this way of looking at
exponentiating. Algorithm 15 Recursive Exponentia-
tion
Input: a, n - Integers a and n ≥ 0

n−1
a ∗ a
 if n > 1
Output: an
an = a if n = 1 1: function rec-exp(a,n)
if n = 0 then return 1

1 if n = 0 2:
3: else if n = 1 then return a
The algorithm implementing the above view of exponentiation is 4: else
as follows. It’s correctness can be proved with simple inductive 5: return a∗rec-exp(a, n−1)
reasoning.

8.2.1 Runtime of recursive exponentiation

Its runtime is something, ok let me tell you this. Say its runtime is T (n) when I input n, (we will discuss
recurrences and their solution in more detail later). We have

1
 if n = 0
T (n) = 1 if n = 1

T (n − 1) if n ≥ 2


8.3 Exponentiation by repeated squaring

We can improve by a lot by considering the following magic of power (or power of magic).

14
  n
/2 n/2
a · a
 if n > 1 even
an = a · an−1/2 · an−1/2 if n is odd Algorithm 16 Exponentiation by re-


1 if n = 0 peated squaring
Input: a, n - Integers a and n ≥ 0
Note that when n is even n/2 is an integer and when n is odd, Output: an
(n−1)/2 is an integer, so in both cases we get the same problem 1: function rep-sq-exp(a,n)
(exponentiating one integer to the power of another integer) but 2: if n = 0 then return 1
of smaller size. And smaller powers are supposed to be easy, 3: else if n > 0 and n is even then
really! well at least n = 0 or 1, or sometime even 2. So we 4: z ← rep-sq-expP(a, n/2)
exploit this formula and use recursion. 5: return z ∗ z
Again correctness of this algorithm follows form the above for- 6: else
mula. But you need to prove it using induction, i.e. prove that 7: z ← rep-sq-exp(a, (n − 1)/2)
this algorithm returns an for all positive integers n, prove the 8: return a ∗ z ∗ z
base case using the stopping condition of this function etc.

8.3.1 Runtime of repeated squaring

It is not very straight-forward to find the runtime of a recursive procedure always, in this case it is quite easy.
Usually the runtime of a recursive algorithm is expressed as a recurrence relation, (that is the time algorithm
takes for an input of size n is defined in terms of the time it takes on inputs of smaller size). In this case we know
that the runtime doesn’t really depends on the size of a (well not directly, we may assume that a is restricted
to be an int. Multiplying y a few times with itself is just like multiplying z a few times with itself, the only
thing that matters is how many times you do the multiplication).

In this case denote the runtime of Rep-Sq-EXP on input a, n 

as R(n) (as discussed above a doesn’t have to be involved in 
 1 if n = 0

1
determining runtime of Rec-EXP). So we have to determine if n = 1
R(n) =
R(n) as a function of n. What we know just by inspecting the 
 R(n/2) + 2 if n > 1 and n is even
algorithm is summarized in the following recurrence relation.

R( /2) + 3 if n > 1 and n is odd
 n−1

We will discuss recurrence relation later, but for now if you think about it as follows: Assume n is a power of 2
so it stays even when halve it.

R(n) = R (n/2) + 2 = R (n/4) + 2 + 2 = R (n/8) + 2 + 2 + 2 = . . .

In general we have
j times
z }| {
R(n) = R (n/2j ) + 2 + 2 + ... + 2

when j = log n, then we get

log n times log n times
z }| { z }| {
R(n) = R (n/2log n ) + 2 + 2 + . . . + 2 = R (n/n) + 2 + 2 + . . . + 2
log n times
z }| {
= R (1) + 2 + 2 + . . . + 2 = 1 + 2 log n

It is very easy to argue that in general that is for n not necessarily power of 2 or even, the runtime is something
like 1 + 3 log n.
Exercise: Give a non-recursive implementation of repeated squaring based exponentiation. You can also use
the binary expansion of n

15
9 Matrix and Vectors Arithmetic
In this section, we develop some algorithmic primitives for arithmetic on matrices and vectors. They are building
blocks for various machine learning and data science solutions.

9.1 Dot Product

Dot Product is an operation defined for two n dimensional vectors. Dot product of two vectors A = (a1 , a2 , . . . , an ),
B = (b1 , b2 , . . . , bn ) is defined as

A B
n
X a1 b1
A·B = ai ∗ bi n
a2 b2 P
i=1
..
· ..
= a i bi
. .
i=1
an bn

Input: Two n-dimensional vectors as arrays A and B

n
P
Output: A · B := ⟨A, B⟩ := A[1]B[1] + . . . + A[n]B[n] := A[i]B[i]
i=1

Dot product is also commonly called an inner product or a scalar product. The a geometric interpretation for
dot product is the following. If u is a unit vector and v is any vector then v · u is the projection of v onto u.
The projection of any point p on v onto u is the point on u closest to p. If v and u are both unit vectors then
v · u is the cos of the angle between the two vectors. So in a way the dot product between two vectors measures
their similarity. It tells us how much of v is in the direction of u.

Algorithm 17 Dot product of two vectors

Input: A, B - n dimensional vectors as ar-
What we’re interested in is how to compute the dot product.
rays of length n
We need to multiply all the corresponding elements and then
Output: s = A · B
sum them up. So we can run a for loop, keep a running sum
1: function dot-prod(A, B)
and at the ith turn add the product of the ith terms to it. The
2: s←0
exact code is given below.
3: for i = i to n do
4: s ← s + A[i] ∗ B[i]
5: return s

Runtime of dot-prod How much time does this take? Well that depends on a lot of factors. What machine
you’re running the program on. How many other programs are being run at the same time. What operating
system you’re using. And many other things. So just asking how much time a program takes to terminate
doesn’t really tell us much. So instead we’ll ask a different question. How many elementary operations does
this program performs. Elementary operations are things such as adding or multiplying two 32 bit numbers,
comparing two 32 bit numbers, swapping elements of an array etc. In particular we’re interested in how the
number of elementary operations performed grows with the size of the input.This captures the efficiency of the
algorithm much better.
So with that in mind, let’s analyze this algorithm. What’s the size of the input? A dot product is always
between two vectors. What can change however is the size of these vectors. So that’s our input size. So suppose
the vectors are both n dimensional. Then this code performs n additions and n multiplications. So a total of
2n elementary operations are performed. Can we do any better? Maybe for particular cases we can, when a
vector contains a lot of zeros. But in the general case probably this is as efficient as we can get.

16
9.2 Matrix-Vector Multiplication
Input: Matrix A and vector b
Output: c = A ∗ b
ˆ Condition: num columns of A = num rows of b
Am×n × bn×1 = Cm×1

Matrix-vector multiplication only for the case when the number of columns in matrix A equals the number of
rows in vector x. So, if A is an m × n matrix (i.e., with n columns), then the product Ax is defined for n × 1
column vectors x. If we let Ax = b, then b is an m × 1 column vector. Matrix vector multiplication should be
known to everyone, it is explained in the following diagram

Algorithm 18 Matrix Vector Multiplication

Dot Product
Input: A, B - m × n matrix and n × 1 vec-
A b c tor respectively given as arrays of appropriate
a11 a12 . . . a1n b1 lengths
a21 a22 . . . a2n b2 Output: C = A × B
a31 a32 . . . a3n = 1: function Mat-VectProd(A, B)
b3
.. .. .. .. .. 2: C[ ][ ] ← zeros(m × 1)
am1 am2 . . . amn bn 3: for i = 1 to m do
m×n n×1 m×1 4: C[i] ← Dot-Prod(A[i][:], B)
return C
ˆ Correct by definition
ˆ Runtime is m dot-products of n-dim vectors
ˆ Total runtime m × n real multiplications and additions

9.3 Matrix Multiplication via dot product

Input: Matrices A and B Output: C = A ∗ B
If A and B are two matrices of dimensions m×n and n×k, then their product is another matrix C of dimensions
m × k such that the (i, j)th entry of C is the dot product of the ith row of A with the jth column of B. That is

(C)ij = Ai · Bj .

This is explained in the following diagram

We know how to compute the dot product of two vectors so we can use that for matrix multiplication. The
code is as follows

17
 b11 . . . b1k
B b21 . . . b2k n × k Algorithm 19 Matrix Matrix Multiplication
.. ..
. . Input: A, B - m × n and n × k matrices respec-
bn1 . . . bnk tively given as arrays of appropriate lengths
Output: C = A × B
Dot Product Dot Product 1: function Mat-MatProd(A, B)
A 2: C[ ][ ] ← zeros(m × k)
a11 a12 . . . a1n c11 . . . 3: for i = 1 to m do
a21 a22 . . . a2n ... 4: for j = 1 to k do
a31 a32 . . . a3n ... C
5: C[i][j] ← dot-prod(A[i][:], B[:][j])
.. .. .. .. ..
. . . . .
am1 am2 . . . amn ... 6: return C
m×n m×k
Analysis : How many elementary operations are performed in this algorithm? i goes from 1 to m and for
each value of i, j goes from 1 to k. So the inner loop runs a total of mk times. Each time the inner loop runs we
compute a dot product of two n dimensional vectors. Computing the dot product takes of two n dimensional
vectors takes 2n operations, so the algorithm uses a total of 2mnk operations. Can we do better for this
problem? We defined matrix multiplication in terms of dot products and we said 2n is the minimum number of
operations needed for a dot product, so it would seem we can’t. But in fact there is a better algorithm for this.
For those interested, you can look up Strassen’s Algorithm for matrix multiplication.

9.4 Matrix-Matrix Multiplication via Matrix-Vector Product

Matrix matrix multiplication (of appropriate dimensions) can be done equivalently through repeated matrix-
vector-multiplication. The process is explained in the following figure followed by its pseudo code.

b11 . . . b1k
B b21 . . . b2k n × k
.. .. Algorithm 20 Matrix Matrix Multiplication
. .

bn1 . . . bnk Input: A, B - m × n and n × k matrices respec-

Matrix-Vector Product
tively given as arrays of appropriate lengths
Output: C = A × B
1: function Mat-MatProd(A, B)
A 2: C[ ][ ] ← zeros(m × k)
a11 a12 . . . a1n c11 . . .
a21 a22 . . . a2n ... 3: for j = 1 to k do
a31 a32 . . . a3n ... C 4: C[:][j] ← Mat-VectProd(A, B[:][j])
..
.
..
.
..
.
..
.
..
. 5: return C
am1 am2 . . . amn ...
m×n m×k
Runtime of this algorithm is k multiplication of dimension m × n matrix with vectors of dimension n × 1. Each
matrix vector multiplication takes mn times as discussed above, so total runtime of this algorithm is kmn.

18