Course Syllabus Template

Course Code / Course Name Introduction to Optimization

Course Instructor Name(s) G. Srinivasaraghavan

Hours Component
4 Lecture (1hr = 1 credit)
Credits (L:T:P)
Tutorial (1hr = 1 credit)
(Lecture : Tutorial : Practical) Practical (2hrs = 1 credit)
L:T:P = 3-1-0 Total Credits = 4
Grading Scheme X 4-point scale (A,A-,B+,B,B-,C+,C,D,F)
(Choose by placing X against
appropriate box) Satisfactory/Unsatisfactory (S / X)
Area of Specialization (if applicable)
(Choose by placing X in box against not more than two areas from the list)
Theory and Systems for Computing Networking and
and Data Communication
Artificial Intelligence and Machine Digital Society
Learning
VLSI Systems Cyber Security
X General Elective
Programme / Branch Course is restricted to the following programmes / branch(es):
(Place X appropriately. More than one is okay)
Programme:
Branch: X CSE
X iMTech X ECE
X M.Tech Digital Society
M.Sc.

Course Category Select one from the following:

(Place X appropriately)
Basic Sciences
CSE Core
ECE Core
CSE Branch Elective
ECE Branch Elective
X Engineering Science and Skills
HSS/M
General

Course Pre-Requisites (Where applicable, state exact course code/name)

First level courses in Algorithms and Programming,

Undergraduate Mathematics with Probability, Linear
Algebra and Calculus.

Template Version 1.1

Template Date April 4, 2021
Additional Focus Areas
Select zero or more from the following and write one sentence explaining the how the focus areas covered as part of
the course.[NAAC criteria 1.1.3, 1.3.2].

Yes /
Focus Area Details
No
Direct focus on employability No
Yes Mathematical / Algorithmic /
Implementation skills required for
formulating and solving real-live
Focus on skill development optimization problems.
Focus on entrepreneurship No
Provides value added / life skills No
(language, writing, communication, etc.)

Course Context and Overview

[Provide introduction to the course]
Optimization is a fundamental ‘horizontal’ across most of the natural and engineering sciences,
computational biology, management, economics, etc. It is central to most problems involving
‘decision making’ of some sort where the intent is to take the ‘best’ decision according some
domain specific criteria. Almost all such problems are naturally modeled as optimization
problems with or without constraints. This course is meant to give a broad introduction to such
problems --- their formulation and solution strategies. The subject is mathematically intense and
course will not avoid the maths involved. However the nature of the subject makes it possible to
often visualize the algorithms while leveraging on our geometric intuition and making use of
appropriate plotting and visualization tools to provide a cushion for the abstraction in the maths.
So implementation will be an integral part of the course, including a substantial project.
Preferred implementation language will be Julia/Python. Broadly the course will focus on
‘breadth’ and not on depth. Many topics which could take a whole course for a comprehensive
discussion, will be restricted to as little as one or two lectures. The intent will be to lay the
conceptual foundations and equip the student with just enough for a deeper dive into the topic if
he/she wishes to, later. The course is expected to be of interest to a wide audience among
students across streams within IIITB.

Scope: Optimization is a very broad term and includes a wide range of algorithms and
techniques and it is not practical to do even do passing justice to all of them in a single self-
contained semester course. This edition of the course focuses largely on optimization in the
‘continuous’ world. Discrete optimization problems would include most of the well known NP-
Hard problems that have been studied extensively in computer science --- Problems on Graphs,,
Network Flow Problems, SAT Family of problems on boolean expressions, Knapsack and Bin-
packing, etc. This course excludes all of these problems, though many of these problems are

Template Version 1.1

Template Date April 4, 2021
solved using ILP methods that will be covered briefly towards the end of the course. Also the
bulk of the course is on unconstrained non-linear optimization with about one third of the course
devoted to constrained optimization --- starting with linear programming and moving on to other
non-linear constrained optimization methods. This choice was based on the following
observations (i) a number of constrained optimization problems can be transformed into
unconstrained ones, (ii) techniques for constrained optimization draw heavily on techniques
developed for unconstrained problems, and lastly (iii) unconstrained optimization for highly non-
linear problems in extremely high dimensions has received tremendous attention of late in
machine learning.

The evaluation will cover both theory and implementation / programming.

Course Outcomes and Competencies

[Course Outcomes are to be stated using appropriate terminology and taxonomy as required by NAAC
and/or NBA. For every course credit, about 2-3 outcomes are recommended. ]

PO/
Id Course Outcome
PSO
CL KC Class (Hrs)

CO1 Understand basic concepts of Optimization and PO1 U C 3

the classes of optimization algorithms P
CO2 Understand the key foundational building blocks PO1 U C 10
for optimization techniques. AP P
CO3 Understand and implement different optimization PO1 U C 32
algorithms. PO4 AP P
PSO4
CO4 Learn to formulate large real-life optimization PO1 U C
problem and implement a strategy to solve them. PO4 AP P
Legend: PO/PSO: Programme Outcomes / Programme Specific Outcomes; CL: Cognitive Level (from Revised
Bloom’s Taxonomy); KC: Knowledge Category (from Revised Bloom’s Taxonomy); Class (Hrs): Number of hours
of instruction; Tut (Hrs): Number of hours of tutorial session (where applicable)

Total Number of hours(Classes) = 45

Template Version 1.1

Template Date April 4, 2021
Concept Map of the Course (Optional)

Course Content
[Provide list-wise topics]

Instruction Schedule
[Provide session-wise schedule]

Each lecture is assumed to be of 90 mins.

Introduction and Mathematical Preliminaries

Lecture 1: Introduction to Optimization Terminology and Problem Formulation; Survey of the classes of
Optimization Techniques

Lectures 2-4: Recap of Relevant Mathematical Concepts --- Vector Spaces & Matrices, (Rank,
Independence, Transformations, Projections, Eigenvalues and Vectors, Norms, ...), Geometry
(Hyperplanes, Convexity, Polytopes), Calculus (Gradients --- uni and multi dimensional).
Gradient Computation --- Numerical Techniques, Automatic Symbolic Differentiation (this is the
backbone of all modern deep learning libraries)

Unidimensional Optimization, Bracketing

Lectures 5-7: Fibonacci Search, Golden Section Search, Quadratic Fit Search, Shubert-Piyavskii Method
(one of the few global optimization methods)

Unconstrained Optimization
Lectures 8-10: Descent Strategies --- Line Search, Trust Region Methods; Termination Criteria
Lecture 11: Newton’s Method and the Levenberg-Marquardt Variant
Lectures 12: Conjugate Gradient Methods
Lectures 13: Quasi Newton Methods
Lectures 14-15: Momentum, Nesterov Variant; Popular Optimization Algorithms in Deep Learning:
RMSProp, Adagrad, Adam, Hypergradient Descent

Global Optimization Methods

Lecture 16: Nelder-Mead Simplex Algorithm, Simulated Annealing

Evolutionary / Population-based Methods for Unconstrained Optimization

Lecture 19: Genetic Algorithms
Lecture 20: Ant-Colony Optimization, Particle Swarm Optimization
Lecture 21: Other methods - Differential Evolution, Firefly Algorithm, Cuckoo Search

Optimization Under Constraints

Template Version 1.1

Template Date April 4, 2021
Lectures 22-23: Dealing with Constraints --- Transformations to Remove Constraints, Lagrange and
Augmented Lagrange Methods --- KKT Conditions, Duality, Interior Point Methods, Other Penalty-based
Methods
Lectures 24-25: Linear Programming --- Simplex Method
Lectures 26-27: Brief Introduction to ILP (Integer Linear Programs) – Unimodular Matrices , Cutting
Plane Methods
Lectures 28-29: Introduction to Convex Optimization and Semi-Definite Programming

Other Possible Topics

These would be taken up depending on the time available --- possibly through student presentations.
Stochastic Methods
Multi-objective Optimization
Multi-disciplinary Optimization
Optimization Under Uncertainty
Surrogate Optimization

Learning Resources
[Mention text books, reference books and other learning resources required as part of the course]

1. Algorithms for Optimization by Mykel J. Kochenderfer and Tim A. Wheeler, The

MIT Press, 2019.

2. An Introduction to Optimization (4th Edition) by Edwin K.P. Chong and Stanislaw

H. Zak, Wiley, 2013.

3. Numerical Optimization (2nd Edition) by Jorge Nocedal and Stephen J. Wright,

Springer, 2006.

4. Optimization for Machine Learning, Ed by Suvrit Sra, Sebastian Nowozin and

Stephen J. Wright, The MIT Press, 2012.

5. For some project ideas: https://aa222.stanford.edu/projects/past-final-projects/ ,

https://www.cs.cmu.edu/~pradeepr/convexopt/past_projects.html ,
http://www.optimisationintherealworld.co.uk/

6. Sources for Benchmark Functions and Datasets:

https://www.sfu.ca/~ssurjano/index.html;
http://infinity77.net/go_2021/scipy_test_functions.html

Template Version 1.1

Template Date April 4, 2021
Assessment Plan
[List grade distribution in terms of % across multiple assessment types (assignments, quizzes, mid-term,
end-term, project, etc.)]
Final grade will be based on weights given below:

30%: Project
30%: Implementation Assignments
20%: Mid-Term Exam (Theory)
20%: End-Term Exam (Theory)

Assignments / Projects
[List exact number of assignments or projects included (provide generic description)]
These are intended to be naive implementations of the algorithms discussed in the course, to give students
a more hands-on feel for the algorithms. It will also give them an appreciation of what it takes for the
implementations to be ‘industry grade’ like the ones in the standard libraries (commercial or otherwise)
that are available. The focus will be on ground-up implementations with minimal use of ready-made
libraries, and the correctness / asymptotic behavior of the implementations. Edge cases, numerical
stability issues and performance considerations will be minimal, if at all.

Projects and Assignments are expected to done in small groups (pairs preferably).

S. No. Focus of Assignment / Project CO Mapping

1 Each group is expected to implement and demonstrate CO1-3
in the class a minimum number from among the
algorithms discussed in the class
2 Each group is expected to pick a project topic at the CO4-8
beginning of the course and work on it through the
semester. Projects presentations will be done at the end
of the semester.

Late Assignment Submission Policy

State any penalty policy for late submission

Unless medically approved excuse, all late submissions are not considered for grading.

Make-up Exam/Submission Policy

State if any specific policy derived from institute policy is applicable. Otherwise leave it as given

As per institute policy

Citation Policy for Papers (if applicable)

[If course includes reading papers and citing them as part of activities, state the citation policy. Mention
“Not applicable” if section is not applicable to the course]

Template Version 1.1

Template Date April 4, 2021
Academic Dishonesty/Plagiarism
State if any specific policy derived from institute policy is applicable. Otherwise leave it as given

As per institute policy

Accommodation of Divyangs
[State any enabling mechanisms for accommodating learners with special needs]

As per institute policy

Template Version 1.1

Template Date April 4, 2021