11/13/23, 8:54 PM Assistants tools - OpenAI API

Documentation API reference Examples Forum Help

Tools Beta

Give Assistants access to OpenAI-hosted tools like Code Interpreter and Knowledge
Retrieval, or build your own tools using Function calling. Usage of OpenAI-hosted tools
comes at an additional fee — visit our help center article to learn more about how these
tools are priced.

The Assistants API is in beta and we are actively working on adding more
functionality. Share your feedback in our Developer Forum!

Code Interpreter
Code Interpreter allows the Assistants API to write and run Python code in a sandboxed
execution environment. This tool can process files with diverse data and formatting, and
generate files with data and images of graphs. Code Interpreter allows your Assistant to
run code iteratively to solve challenging code and math problems. When your Assistant
writes code that fails to run, it can iterate on this code by attempting to run different code
until the code execution succeeds.

Enabling Code Interpreter

Pass the code_interpreter in the tools parameter of the Assistant object to enable
Code Interpreter:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 assistant = client.beta.assistants.create(
2 instructions="You are a personal math tutor. When asked a math question, wr
3 model="gpt-4-1106-preview",
4 tools=[{"type": "code_interpreter"}]
5 )

https://platform.openai.com/docs/assistants/tools/supported-files 1/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

The model then decides when to invoke Code Interpreter in a Run based on the nature of
the user request. This behavior can be promoted by prompting in the Assistant's
instructions (e.g., “write code to solve this problem”).

Passing files to Code Interpreter

Code Interpreter can parse data from files. This is useful when you want to provide a large
volume of data to the Assistant or allow your users to upload their own files for analysis.

Files that are passed at the Assistant level are accessible by all Runs with this Assistant:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 # Upload a file with an "assistants" purpose

2 file = client.files.create(
3 file=open("speech.py", "rb"),
4 purpose='assistants'
5 )
6
7 # Create an assistant using the file ID
8 assistant = client.beta.assistants.create(
9 instructions="You are a personal math tutor. When asked a math question, w
10 model="gpt-4-1106-preview",
11 tools=[{"type": "code_interpreter"}],
12 file_ids=[file.id]
13 )

Files can also be passed at the Thread level. These files are only accessible in the specific
Thread. Upload the File using the File upload endpoint and then pass the File ID as part of
the Message creation request:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 thread = client.beta.threads.create(
2 messages=[
3 {
4 "role": "user",
5 "content": "I need to solve the equation `3x + 11 = 14`. Can you help m
6 "file_ids": [file.id]
7 }
L

https://platform.openai.com/docs/assistants/tools/supported-files 2/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

8 ]
9 )

Files have a maximum size of 512 MB. Code Interpreter supports a variety of file formats
including .csv , .pdf , .json and many more. More details on the file extensions (and
their corresponding MIME-types) supported can be found in the Supported files section
below.

Reading images and files generated by Code Interpreter

Code Interpreter in the API also outputs files, such as generating image diagrams, CSVs,
and PDFs. There are two types of files that are generated:

1 Images
2 Data files (e.g. a csv file with data generated by the Assistant)

When Code Interpreter generates an image, you can look up and download this file in the
file_id field of the Assistant Message response:

1 {
2 "id": "msg_OHGpsFRGFYmz69MM1u8KYCwf",
3 "object": "thread.message",
4 "created_at": 1698964262,
5 "thread_id": "thread_uqorHcTs46BZhYMyPn6Mg5gW",
6 "role": "assistant",
7 "content": [
8 {
9 "type": "image_file",
10 "image_file": {
11 "file_id": "file-WsgZPYWAauPuW4uvcgNUGcb"
12 }
13 }
14 ]
15 # ...
16 }

The file content can then be downloaded by passing the file ID to the Files API:

https://platform.openai.com/docs/assistants/tools/supported-files 3/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

content = client.files.retrieve_content(file.id)

When Code Interpreter references a file path (e.g., ”Download this csv file”), file paths are
listed as annotations. You can convert these annotations into links to download the file:

1 {
2 "id": "msg_3jyIh3DgunZSNMCOORflDyih",
3 "object": "thread.message",
4 "created_at": 1699073585,
5 "thread_id": "thread_ZRvNTPOoYVGssUZr3G8cRRzE",
6 "role": "assistant",
7 "content": [
8 {
9 "type": "text",
10 "text": {
11 "value": "The rows of the CSV file have been shuffled and
12 "annotations": [
13 {
14 "type": "file_path",
15 "text": "sandbox:/mnt/data/shuffled_file.csv",
16 "start_index": 167,
17 "end_index": 202,
18 "file_path": {
19 "file_id": "file-oSgJAzAnnQkVB3u7yCoE9CBe"
20 }
21 }
22 ...

Input and output logs of Code Interpreter

By listing the steps of a Run that called Code Interpreter, you can inspect the code input
and outputs logs of Code Interpreter:

python Copy
L

https://platform.openai.com/docs/assistants/tools/supported-files 4/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 run_steps = client.beta.threads.runs.steps.list(
2 thread_id=thread.id,
3 run_id=run.id
4 )

1 {
2 "object": "list",
3 "data": [
4 {
5 "id": "step_DQfPq3JPu8hRKW0ctAraWC9s",
6 "object": "thread.run.step",
7 "type": "tool_calls",
8 "run_id": "run_kme4a442kme4a442",
9 "thread_id": "thread_34p0sfdas0823smfv",
10 "status": "completed",
11 "step_details": {
12 "type": "tool_calls",
13 "tool_calls": [
14 {
15 "type": "code",
16 "code": {
17 "input": "# Calculating 2 + 2\nresult = 2 + 2\nresul
18 "outputs": [
19 {
20 "type": "logs",
21 "logs": "4"
22 }
23 ...
24 }

Knowledge Retrieval
Retrieval augments the Assistant with knowledge from outside its model, such as
proprietary product information or documents provided by your users. Once a file is
uploaded and passed to the Assistant, OpenAI will automatically chunk your documents,
L

https://platform.openai.com/docs/assistants/tools/supported-files 5/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

index and store the embeddings, and implement vector search to retrieve relevant content
to answer user queries.

Enabling Retrieval
Pass the retrieval in the tools parameter of the Assistant to enable Retrieval:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 assistant = client.beta.assistants.create(
2 instructions="You are a customer support chatbot. Use your knowledge base t
3 model="gpt-4-1106-preview",
4 tools=[{"type": "retrieval"}]
5 )

How it works
The model then decides when to retrieve content based on the user Messages. The
Assistants API automatically chooses between two retrieval techniques:

1 it either passes the file content in the prompt for short documents, or
2 performs a vector search for longer documents

Retrieval currently optimizes for quality by adding all relevant content to the context of
model calls. We plan to introduce other retrieval strategies to enable developers to choose
a different tradeoff between retrieval quality and model usage cost.

Uploading files for retrieval

Similar to Code Interpreter, files can be passed at the Assistant-level or at the Thread-level

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 # Upload a file with an "assistants" purpose

2 file = client.files.create(
3 file=open("knowledge.pdf", "rb"),
4 purpose='assistants'
5 )
L 6

https://platform.openai.com/docs/assistants/tools/supported-files 6/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

7 # Add the file to the assistant

8 assistant = client.beta.assistants.create(
9 instructions="You are a customer support chatbot. Use your knowledge base
10 model="gpt-4-1106-preview",
11 tools=[{"type": "retrieval"}],
12 file_ids=[file.id]
13 )

Files can also be added to a Message in a Thread. These files are only accessible within
this specific thread. After having uploaded a file, you can pass the ID of this File when
creating the Message:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 message = client.beta.threads.messages.create(
2 thread_id=thread.id,
3 role="user",
4 content="I can't find in the PDF manual how to turn off this device.",
5 file_ids=[file.id]
6 )

Maximum file size is 512MB. Retrieval supports a variety of file formats including .pdf ,
.md , .docx and many more. More details on the file extensions (and their
corresponding MIME-types) supported can be found in the Supported files section below.

Deleting files
To remove a file from the assistant, you can detach the file from the assistant:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 file_deletion_status = client.beta.assistants.files.delete(
2 assistant_id=assistant.id,
3 file_id=file.id
4 )

Detaching the file from the assistant removes the file from the retrieval index as well.

https://platform.openai.com/docs/assistants/tools/supported-files 7/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

File citations

When Code Interpreter outputs file paths in a Message, you can convert them to
corresponding file downloads using the annotations field. See the Annotations section
for an example of how to do this.

1 {
2 "id": "msg_3jyIh3DgunZSNMCOORflDyih",
3 "object": "thread.message",
4 "created_at": 1699073585,
5 "thread_id": "thread_ZRvNTPOoYVGssUZr3G8cRRzE",
6 "role": "assistant",
7 "content": [
8 {
9 "type": "text",
10 "text": {
11 "value": "The rows of the CSV file have been shuffled an
12 "annotations": [
13 {
14 "type": "file_path",
15 "text": "sandbox:/mnt/data/shuffled_file.csv",
16 "start_index": 167,
17 "end_index": 202,
18 "file_path": {
19 "file_id": "file-oSgJAzAnnQkVB3u7yCoE9CBe"
20 }
21 }
22 ]
23 }
24 }
25 ],
26 "file_ids": [
27 "file-oSgJAzAnnQkVB3u7yCoE9CBe"
28 ],
29 ...
30 },

https://platform.openai.com/docs/assistants/tools/supported-files 8/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

Function calling
Similar to the Chat Completions API, the Assistants API supports function calling.
Function calling allows you to describe functions to the Assistants and have it intelligently
return the functions that need to be called along with their arguments. The Assistants API
will pause execution during a Run when it invokes functions, and you can supply the
results of the function call back to continue the Run execution.

Defining functions

First, define your functions when creating an Assistant:

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 assistant = client.beta.assistants.create(
2 instructions="You are a weather bot. Use the provided functions to answer
3 model="gpt-4-1106-preview",
4 tools=[{
5 "type": "function",
6 "function": {
7 "name": "getCurrentWeather",
8 "description": "Get the weather in location",
9 "parameters": {
10 "type": "object",
11 "properties": {
12 "location": {"type": "string", "description": "The city and state
13 "unit": {"type": "string", "enum": ["c", "f"]}
14 },
15 "required": ["location"]
16 }
17 }
18 }, {
19 "type": "function",
20 "function": {
21 "name": "getNickname",
22 "description": "Get the nickname of a city",
23 "parameters": {
24 "type": "object",
25 "properties": {
26 "location": {"type": "string", "description": "The city and state
27 },
28 "required": ["location"]
29 }
L 30 }

https://platform.openai.com/docs/assistants/tools/supported-files 9/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

31 }]
32 )

Reading the functions called by the Assistant

When you initiate a Run with a user Message that triggers the function, the Run will enter a
pending status. After it processes, the run will enter a requires_action state which
you can verify by retrieving the Run. The model can provide multiple functions to call at
once using parallel function calling:

1 {
2 "id": "run_3HV7rrQsagiqZmYynKwEdcxS",
3 "object": "thread.run",
4 "assistant_id": "asst_rEEOF3OGMan2ChvEALwTQakP",
5 "thread_id": "thread_dXgWKGf8Cb7md8p0wKiMDGKc",
6 "status": "requires_action",
7 "required_action": {
8 "type": "submit_tool_outputs",
9 "submit_tool_outputs": {
10 "tool_calls": [
11 {
12 "id": "call_Vt5AqcWr8QsRTNGv4cDIpsmA",
13 "type": "function",
14 "function": {
15 "name": "getCurrentWeather",
16 "arguments": "{\"location\":\"San Francisco\"}"
17 }
18 },
19 {
20 "id": "call_45y0df8230430n34f8saa",
21 "type": "function",
22 "function": {
23 "name": "getNickname",
24 "arguments": "{\"location\":\"Los Angeles\"}"
25 }
26 }
27 ]
28 }
L

https://platform.openai.com/docs/assistants/tools/supported-files 10/12
11/13/23, 8:54 PM Assistants tools - OpenAI API

29 },
30 ...

Submitting functions outputs

You can then complete the Run by submitting the tool output from the function(s) you call.
Pass the tool_call_id referenced in the required_action object above to match
output to each function call.

python Copy

Upgrade to Python SDK v1.2 with pip install --upgrade openai

1 run = client.beta.threads.runs.submit_tool_outputs(
2 thread_id=thread.id,
3 run_id=run.id,
4 tool_outputs=[
5 {
6 "tool_call_id": call_ids[0],
7 "output": "22C",
8 },
9 {
10 "tool_call_id": call_ids[1],
11 "output": "LA",
12 },
13 ]
14 )

After submitting outputs, the run will enter the queued state before it continues it’s
execution.

Supported files
For text/ MIME types, the encoding must be one of utf-8 , utf-16 , or ascii .

FILE CODE
FORMAT MIME TYPE INTERPRETER RETRIEVAL

.c text/x-c

.cpp text/x-c++

.csv application/csv
L

https://platform.openai.com/docs/assistants/tools/supported-files 11/12
11/13/23, 8:54 PM Assistants tools - OpenAI API
FILE CODE
FORMAT MIME TYPE INTERPRETER RETRIEVAL

.docx application/vnd.openxmlformats-
officedocument.wordprocessingml.document

.html text/html

.java text/x-java

.json application/json

.md text/markdown

.pdf application/pdf

.php text/x-php

.pptx application/vnd.openxmlformats-
officedocument.presentationml.presentation

.py text/x-python

.py text/x-script.python

.rb text/x-ruby

.tex text/x-tex

.txt text/plain

.css text/css

.jpeg image/jpeg

.jpg image/jpeg

.js text/javascript

.gif image/gif

.png image/png

.tar application/x-tar

.ts application/typescript
L

https://platform.openai.com/docs/assistants/tools/supported-files 12/12