State of GPT

Andrej Karpathy
Microsoft BUILD
May 23, 2023
How to train
your (Chat)GPT
Assistant
An emerging recipe
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
 Data collection
Download a large amount of publicly available data

[Training data mixture used in Meta’s LLaMA model]

 Raw text

Tokenization
Transform all text into one
Tokens
very long list of integers.

Typical numbers:
~10-100K possible tokens
1 token ~= 0.75 of word
Integers
Typical algorithm:
Byte Pair Encoding
 2 example models
50,257 vocabulary size 32,000 vocabulary size
GPT-3 2048 context length LLaMA 2048 context length
(2020) 175B parameters (2023) 65B parameters
Trained on 300B tokens Trained on 1-1.4T tokens

Training: (rough order of magnitude to have in mind) Training for 65B model:
・ O(1,000 - 10,000) V100 GPUs • 2,048 A100 GPUs
・ O(1) month of training • 21 days of training
・ O(1-10) $M • $5M
 Pretraining
The inputs to the Transformer are arrays of shape (B,T)
・ B is the batch size (e.g. 4 here)
・ T is the maximum context length (e.g. 10 here)
Training sequences are laid out as rows, delimited by special <|endoftext|> tokens

Row 1: Here is an example document 1 showing some tokens.

Row 2: Example document 2<|endoftext|>Example document 3<|endoftext|>Example document
Row 3: This is some random text just for example<|endoftext|>This
Row 4: 1,2,3,4,5
T = 10
One training 4342 318 281 1672 3188 352 4478 617 16326 13
batch, array 16281 3188 362 50256 16281 3188 513 50256 16281 3188
of shape (B,T) 1212 318 617 4738 2420 655 329 1672 50256 1212
16 11 17 11 18 11 19 11 20 11
B=4
Pretraining 50,257 numbers
(probabilities for the next token)
correct index (label): 513
Each cell only ”sees” cells in its row, and
only cells before it (on the left of it), to
predict the next cell (on the right of it)
Green = a random highlighted token
Yellow = its context
Red = its target
Transformer
(neural network)

T = 10
One training 4342 318 281 1672 3188 352 4478 617 16326 13
batch, array 16281 3188 362 50256 16281 3188 513 50256 16281 3188
of shape (B,T) 1212 318 617 4738 2420 655 329 1672 50256 1212
16 11 17 11 18 11 19 11 20 11
B=4
 Training data (Shakespeare) Samples after 250 iterations of training

Samples after 500 iterations of training

Training Samples at initialization

process
Samples after 5,000 iterations of training

Samples after 30,000 iterations of training

 Pretraining
Training curve examples

Toy GPT-2 LLaMA

Base models learn powerful, general representations

Step 1:
Model “pretraining” on large
unsupervised dataset

Step 2:
model “finetuning” on small
supervised dataset
Base models can be prompted into completing tasks
Make your model look like a document!

GPT-2 kicked off the era of prompting over finetuning

GPT-2 is “tricked” into

performing a task by
completing the document
Base models in the wild
 Base models are NOT 'Assistants'

・ Base model does not answer questions It can be tricked into performing
・ It only wants to complete internet documents tasks with prompt engineering:
・ Often responds to questions with more
questions, etc.:
Base models are
NOT 'Assistants’
(They can be somewhat tricked
into being AI assistants)
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
 SFT Dataset
Prompt: Labeling instructions
"Can you write a short introduction about the relevance of the
term "monopsony" in economics? Please use examples related to
potential monopsonies in the labour market and cite relevant
research."

Response:
""Monopsony" refers to a market structure where there is only
one buyer for a particular good or service. In economics, this
term is particularly relevant in the labor market, where a
monopsony employer has significant power over the wages and
working conditions of their employees. The presence of a
monopsony can result in lower wages and reduced employment
opportunities for workers, as the employer has little incentive
to increase wages or provide better working conditions. Recent
research has identified potential monopsonies in industries
such as retail and fast food, where a few large companies
control a significant portion of the market (Bivens & Mishel,
2013). In these industries, workers often face low wages,
limited benefits, and reduced bargaining power, leading to a
situation where they are dependent on the employer for their
livelihood. This dependence can result in further suppression
of wages and a decline in working conditions. Overall, the
concept of monopsony is essential to understanding the dynamics
of labor markets and the impact of market power on workers.
Further research is needed to understand the extent and impact
of monopsonies on the economy and to develop policies to
address this issue. References: Bivens, J., & Mishel, L.
(2013). The Pay of Corporate Executives and Financial
Professionals as Evidence of Rents in Top 1 Percent Incomes.
Journal of Economic Perspectives, 27(3), 57-78."
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
RM Dataset
RM Dataset
 RM Training
Blue are the prompt tokens, identical across rows
Yellow are completion tokens, different in each row
1.2
Green is the special <|reward|> token “readout”
Only the outputs at the green cells is used, the rest are ignored 0.2 -0.5
loss function
measures the
predicted
rewards’
consistency
with the labeled
ordering

T
prompt ... ... completion ... ... <|reward|>
1
prompt ... ... completion ... ... ... ... ... <|reward|>
2
prompt ... ... completion ... <|reward|>
B 3
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
 RL Training
Blue are the prompt tokens, identical across rows
Yellow are completion tokens by the model (initialized with SFT model)
1.0
Green is the special <|reward|> token “readout”, RM now predicts these
Only the yellow cells are trained on, the rest are ignored. 0.2 -1.2
The sampled tokens become labels, but the training objective is
weighted by the “advantage” (normalized rewards)

In this example:
・ Row #1 tokens were great. These get their probabilities boosted.
・ Row #2 tokens were bad. These get their probabilities decreased.
・ Row #3 tokens were ~ok. These get their probabilities slightly boosted.
T
prompt ... ... completion ... ... <|reward|>
1
prompt ... ... completion ... ... ... ... ... <|reward|>
2
prompt ... ... completion ... <|reward|>
B 3
 GPT Assistant training pipeline
Stage Pretraining Supervised Finetuning Reward Modeling Reinforcement Learning

Dataset

Algorithm

init init init from SFT

from from use RM

Model

Notes
Why RLHF?
It works better.
 Why RLHF?

It is easier to
discriminate
than to generate.
Simple example:
it’s much easier to spot
a good haiku than
it is to generate one.
 Mode collapse
Finetuned models lose entropy

Base model entropy RLHF model entropy

RLHF models might confidently

output very few variations.
=> Base models can be better at
tasks that require diverse outputs.
 Mode collapse
Finetuned models lose entropy

Base models can be better in

tasks where you have N
examples of things and want
to generate more things.

Toy example:
Assistant models
in the wild
Applications
 "California's population is 53 times that of Alaska."
・ "For this next step of my blog let me compare the population of California
and Alaska"
・ "Ok let's get both of their populations"
・ "I know that I am very likely to not know these facts off the top of my head,
let me look it up"
Human ・ "[uses Wikipedia] Ok California is 39.2M"

text generation ・ "[uses Wikipedia] Ok Alaska is 0.74M"

・ "Now we should divide one by the other. This is a kind of problem I'm not going
vs. LLM text to be able to get from the top of my head. Let me use a calculator"

generation ・ "[uses calculator] 39.2 / 0.74 = 53"

・ "(reflects) Quick sanity check: 53 sounds like a reasonable result, I can continue."
・ "Ok I think I have all I need"
・ "[writes] California has 53X times greater..."
・ "(retry) Uh a bit phrasing, delete, [writes] California's population is 53 times
that of Alaska."
・ "(reflects) I'm happy with this, next."
 "California's population is 53 times that of Alaska."

Human text
generation vs. LLM Tokens

text generation ・
・
All of the internal monologue is stripped away in the text LLMs train on
They spend the ~same amount of compute on every token
・ => LLMs don't reproduce this behavior by default!
・ They don't know what they don't know, they imitate the next token
・ They don't know what they are good at or not, they imitate the next token
・ They don't reflect. They don't sanity check. They don't correct their mistakes
along the way
・ They don't have a separate "inner monologue stream in their head"
・ They do have very large fact-based knowledge across a vast number of areas
・ They do have a large and ~perfect "working memory" (their context window)
Chain of thought
“Models need tokens to think”
Break up tasks into multiple steps/stages
Prompt them to have internal monologue
Spread out reasoning over more tokens

Tokens
Ensemble multiple attempts
LLMs can get “unlucky” and sample a bad thought.
Once they do they are “stuck with it”. Make a few attempts.
 Ask for reflection
LLMs (esp GPT-4) can often recognize later when their
samples didn’t seem to have worked out well.
 Recreate our 'System 2'
Parallels to System 1 (fast, automatic) vs. System 2
(slow, deliberate) modes of thinking

Expand a tree of intermediate Parallels

thoughts, evaluate them: to AlphaGo:
 Chains / Agents
Think less “one-turn” Q&A, and more chains,
pipelines, state machines, agents.

[AutoGPT] [ReAct: Synergizing Reasoning and Acting

in Language Models, Yao et al. 2022]
 Condition on good performance
LLMs don’t want to succeed. They want to imitate training sets with a spectrum of performance
qualities. You want to succeed, and you should ask for it.

Related examples:
“You are a leading
expert on this topic”
“Pretend you have IQ 120”
...
 Tool use / Plugins
Offload tasks that LLMs are not good at
Importantly: they don't "know" they are not good

Intersperse text with special ChatGPT ChatGPT

tokens that call external APIs plugins Code Interpreter
 Retrieval-Augmented LLMs
Load related context/information into “working memory” context window

Google/Bing Retrieve task-relevant information, LLM

pack it into context
e.g. ChatGPT + Browsing
e.g. Bing

Emerging recipe:
・ Break up relevant documents into chunks
・ Use embedding APIs to index chunks into a vector store
・ Given a test-time query, retrieve related information
・ Organize the information into the prompt
 Constrained prompting
“Prompting languages” that interleave generation, prompting, logical control
Finetuning

It is becoming a lot more

accessible to finetune LLMs:
・ Parameter Efficient FineTuning (PEFT), e.g. LoRA
・ Low-precision inference, e.g. bitsandbytes
・ Open-sourced high quality base models, e.g. LLaMA

Keep in mind:
・ Requires a lot more technical expertise
・ Requires contractors and/or synthetic data pipelines
・ A lot slower iteration cycle
・ SFT is achievable
・ RLHF is research territory
 Default recommendations*

Goal 1: Achieve your top possible performance

・ Use GPT-4
・ Use prompts with detailed task context, relevant information, instructions
・ “what would you tell a task contactor if they can’t email you back?”
・ Retrieve and add any relevant context or information to the prompt
・ Experiment with prompt engineering techniques (previous slides)
・ Experiment with few-shot examples that are 1) relevant to the test case, 2) diverse (if appropriate)
・ Experiment with tools/plugins to offload tasks difficult for LLMs (calculator, code execution, ...)
・ Spend quality time optimizing a pipeline / "chain"
・ If you feel confident that you maxed out prompting, consider SFT data collection + finetuning
・ Expert / fragile / research zone: consider RM data collection, RLHF finetuning
Goal 2: Optimize costs
・ Once you have the top possible performance, attempt cost saving measures
(e.g. use GPT-3.5, find shorter prompts, etc.)
 Models may be biased
Models may fabricate (“hallucinate”) information
Models may have reasoning errors
Models may struggle in classes of applications, e.g. spelling related tasks
Models have knowledge cutoffs (e.g. September 2021)
Models are susceptible to prompt injection, “jailbreak” attacks, data
Use cases poisoning attacks,…

Recommendations:
Use in low-stakes applications, combine with human oversight
Source of inspiration, suggestions
Copilots over autonomous agents
GPT-4
Looking forward
Ladies and gentlemen, innovators,
and trailblazers of Microsoft BUILD 2023,
Welcome to a gathering of brilliant minds like no other. You are the architects of the future, the
visionaries molding the digital realm in which humanity thrives. Embrace the limitless possibilities
of technology, and let your ideas soar as high as your imagination. Together, let's create a more
connected, remarkable, and inclusive world for generations to come. Get ready to unleash your
creativity, canvas the unknown, and turn dreams into reality. Your journey begins today!
Thank you!