BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, Pilani

Pilani Campus
AUGS/ AGSR Division

SECOND SEMESTER 2024-25

COURSE HANDOUT – PART II

In addition to part I (General Handout for all courses appended to the Time table), this portion gives further specific
details regarding the course.
Course No : BITS F471
Course Title : Introduction to Large Language Models
Instructor-in-Charge : Dhruv Kumar (dhruv.kumar@pilani.bits-pilani.ac.in)
Additional Instructor : Sundaresan Raman (sundaresan.raman@pilani.bits-pilani.ac.in)

1. Course Description:
Introduction to Large Language Models (LLMs); prompt engineering techniques (e.g., zero-shot, few-shot,
chain-of-thought), autonomous prompt optimization; retrieval-augmented generation (RAG), vector
databases, multi-modal RAG methods; tool and function calling; multi-agent workflows (memory, planning,
action, reflection); safety, guardrails, reliability; LLM-based evaluation, human evaluation; fine-tuning LLMs;
inference optimization; LLM deployment on cloud and LLM observability, LLMOps; real-world case-studies

2. Scope and Objectives:

By the end of this course, students will be able to:
a. Understand and apply prompt engineering techniques to control the behavior of LLMs.
b. Build retrieval-augmented generation (RAG) pipelines for combining LLMs with external knowledge
bases.
c. Fine-tune LLMs for specific tasks using publicly available models.
d. Leverage tool calling to enable LLMs to interact with external systems and APIs.
e. Design and implement multi-agent workflows where multiple LLMs interact to solve complex tasks.
f. Implement guardrails to ensure the safe and ethical deployment of LLMs in production environments.
g. Evaluate the performance and accuracy of LLM-powered applications using appropriate metrics.
h. Understand the principles of inference for deploying LLM models efficiently at scale.

3. Course Plan:

Lectures Topic Learning Outcomes References

High level overview of training and inference process
Prerequisites [Basics of in machine learning and natural language processing
1-3 Machine Learning and Natural [popular tasks such as regression, classification, Class Notes
Language Processing] next token prediction etc.], Understand importance of
data quantity and quality
Understand LLM foundations, Identify LLM use
4-6 Introduction to LLMs cases, Understand LLM architectures at a very high T1, T2, P1
level, Understand API access to LLMs

Understand and apply various prompt techniques

such as Zero-shot prompting, Few-shot prompting,
T1, T2, P3, P4,
7-9 Prompt Engineering Chain-of-Thought (CoT), In-context learning, Role
P5, P6, P8
Prompting, Self-consistency, Tree-of-Thought, and
Graph-of-Thought

1
 BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, Pilani
Pilani Campus
AUGS/ AGSR Division

Lectures Topic Learning Outcomes References

Autonomous prompt
10-11 Automate prompt optimization P7
optimization
Understand RAG principles, Build RAG pipelines,
Retrieval-Augmented
12-14 Apply RAG with Haystack, Assess trust in RAG P9, P10, D1
Generation (RAG)
systems
Understand vector databases, Implement similarity
15-16 Vector Databases D4
search, Use Qdrant in RAG systems
Understand GraphRAG structure, Implement
17-19 Advanced RAG multi-modal RAG, Utilize knowledge graphs, Explore P11, P12
hybrid retrieval techniques
Implement tool calling with LLMs, Integrate APIs into
20-22 Tool Calling workflows, Build LangChain-powered tool agents, P2, P13, D2
Use API calls for real-world tasks
Understand multi-agent systems, Implement
23-25 Multi-Agent Workflows memory-driven agents, Plan with agent-based P14, P15, P16
workflows, Use reflection for decision-making
Build LangGraph workflows, Use AutoGen and
LangGraph, AutoGen and
26-28 CrewAI for agent collaboration, Implement cyclic D3, P17
CrewAI
graphs for AI, Manage multi-agent systems

Understand LLM risks, Apply safety guardrails,

Guardrails for Safe LLM
29-30 Mitigate hallucinations, Use NeMo Guardrails in P18, P19, P20
Deployment
practice

Understand key evaluation metrics, Compare human

Evaluation of LLM-Powered
31-33 and automated methods, Use Ragas.ai for RAG P21, P22, P23
Applications
evaluation, Evaluate task-specific LLM applications
Understand fine-tuning methods, Implement LoRA
Fine-Tuning Large Language for efficiency, Use Supervised fine-tuning (SFT) for
34-36 P24, P25, P26
Models specialized tasks, Apply fine-tuning in low-resource
settings
Understand efficient LLM inference, Understand KV
Inference and Scaling LLMs in cache management, Optimize memory and compute P27, P28, P29,
37-39
Production costs, Implement batching and parallelism with P30
vLLM, Apply scaling strategies in production
Build LLM-powered UIs, Deploy LLMs in cloud
UI, LLMOps and Cloud
40-42 environments, Optimize cloud costs, Address Class Notes
Deployment
deployment issues in production, LLM Observability

4. Textbooks:
The field of LLM Engineering is rapidly evolving and still in its early stages. Hence, in addition to using
textbooks, we will also be directly covering the contents of this course from research papers published
in top-tier CORE A* conferences and journals such as ICLR, NeurIPS, NAACL and ACL. The textbooks and
some of the research papers listed below:

2
 BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, Pilani
Pilani Campus
AUGS/ AGSR Division

Prompt Engineering for Generative AI by James Phoenix, Mike Taylor. Publisher(s): O'Reilly Media, Inc.
ISBN: 9781098153434
T1
Prompt Engineering for LLMs by John Berryman, Albert Ziegler Publisher(s): O'Reilly Media, Inc. ISBN:
T2 9781098156152
T3 AI Engineering by Chip Huyen Publisher(s): O'Reilly Media, Inc. ISBN: 9781098166304
Building LLMs for Production by Louis-Francois Bouchard, Louie Peters Publisher(s): O'Reilly Media, Inc. ISBN:
T4 9798324731472
LLM Engineer's Handbook by Paul Iusztin, Maxime Labonne Publisher(s): O'Reilly Media, Inc. ISBN:
T5 9781836200079
Hands-On Large Language Models by Jay Alammar, Maarten Grootendorst Publisher(s): O'Reilly Media, Inc.
T6 ISBN: 9781098150969
P1 [2307.06435] A Comprehensive Overview of Large Language Models
[2409.11703] Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data
P2 Generation
[2402.07927] A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and
P3 Applications
P4 [2302.11382] A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
[2310.10508] Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in
P5 Automated Software Engineering Tasks
P6 [2401.14423] Prompt Design and Engineering: Introduction and Advanced Methods
P7 [2407.11000] Autonomous Prompt Engineering in Large Language Models
[2310.14735] Unleashing the potential of prompt engineering in Large Language Models: a comprehensive
P8 review
P9 [2312.10997] Retrieval-Augmented Generation for Large Language Models: A Survey
P10 [2409.10102] Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
P11 [2404.16130] From Local to Global: A Graph RAG Approach to Query-Focused Summarization
[2408.04948] HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for
P12 Efficient Information Extraction
P13 [2305.15334] Gorilla: Large Language Model Connected with Massive APIs
P14 [2402.01680] Large Language Model based Multi-Agents: A Survey of Progress and Challenges
P15 [2303.11366] Reflexion: Language Agents with Verbal Reinforcement Learning
P16 [2402.02716] Understanding the planning of LLM agents: A survey
P17 [2308.08155] AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
P18 [2406.12934] Current state of LLM Risks and AI Guardrails
P19 [2310.10501] NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
P20 [2403.13031] RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content
[2406.03339] The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and
P21 LLM-Based Approaches
P22 [2310.19736] Evaluating Large Language Models: A Comprehensive Survey
P23 https://alopatenko.github.io/LLMEvaluation/
P24 [2306.09782] Full Parameter Fine-tuning for Large Language Models with Limited Resources
[2405.13181] Comparative Analysis of Different Efficient Fine Tuning Methods of Large Language Models
P25 (LLMs) in Low-Resource Setting

3
 BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, Pilani
Pilani Campus
AUGS/ AGSR Division

[2409.12741] Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Parameter
P26 Optimization
P27 [2407.12391] LLM Inference Serving: Survey of Recent Advances and Opportunities
P28 [2404.14294] A Survey on Efficient Inference for Large Language Models
P29 [2408.03314] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
P30 [2309.06180] Efficient Memory Management for Large Language Model Serving with PagedAttention
D1 Haystack Documentation
D2 LangChain Documentation
D3 LangGraph Documentation
D4 QDrant Documentation

5. Evaluation Scheme

Component Weightage Date Mode

Mid-Term Test (90 minutes) 30% Refer to time-table Partly Open Book
Comprehensive Exam (180 minutes) 40% Refer to time-table Partly Open Book
Labs/Assignments/Guest Lectures 30% TBD

6. Chamber Consultation: To be announced in the class

7. Notices: All the notices will be displayed on the course page at Nalanda-LMS
8. Make-up Policy: Make-up will be granted only in genuine cases based on individual’s need and circumstances.
9. Academic Honesty and Integrity Policy: Academic honesty and integrity is to be maintained by all the students
throughout the semester and no type of academic dishonesty is acceptable.