Lesson · 14 min

Tokens and limits

What a token is

A model does not read letters or words. The service first cuts the text into tokens. A token is a short piece of text: a word, a part of a word, or a punctuation mark. Each service has its own rules for the cut. With many services, 100 tokens are about 75 words of English text.

The simulated service of this course has a simple rule. Each word is one token, and each punctuation mark is one token. The function count_tokens applies the rule.

from sim_model import count_tokens

count_tokens("The dock was full.")   # 5

The result is 5: four words and the period.

A real service gives you a tool that counts the tokens for its models. Use that tool when the number is important. Do not calculate the number from the characters of the text.

Why tokens are important

Price. You pay for each token that goes in, and for each token that comes out. Lesson 4 calculates the price.
Limits. A model can work with a fixed number of tokens in one call.
Time. A model writes its answer one token after the other. A long answer takes more time.

The limit of the input

Each model has a context window. That is the largest number of tokens that the model can work with in one call. The dictionary MODELS has the limit of each simulated model.

from sim_model import MODELS

MODELS["sim-small"]["context"]   # 300
MODELS["sim-large"]["context"]   # 1500

These numbers are small, so that you can reach the limit in a lesson. A real model has a much larger window, often 100,000 tokens or more.

The simulated service compares the tokens of all messages with the limit. Most real services count the messages and the answer together.

When the messages are too long, the service does not answer. It refuses the call with an error such as this one.

The messages have 412 tokens. The limit of sim-small is 300.

A program that sends long text counts the tokens before the call. Then it can make the text shorter, or it can use a model with a larger window.

The limit of the output

The parameter max_tokens sets the largest number of tokens that the model can write.

response = client.chat(messages, max_tokens=5)

The model does not make its answer fit the limit. It writes its usual answer, and the service stops it at the limit. The key stop_reason of the response tells you what happened.

`stop_reason`	What it means
`"end"`	The model finished its answer.
`"max_tokens"`	The answer reached the limit, and the service cut it.

A cut answer can end in the middle of a sentence. Read the stop reason before you use an answer.

A limit protects you from a long answer that costs much. A limit that is too low cuts good answers. Choose a limit that has room for the longest answer that you expect.

Your turn

main.py has instructions and a comment. Do the tasks in order.

Tasks

5 to do

Task 1, not checked yet. Make a variable comment_tokens with the number of tokens in comment. Use count_tokens.
Task 2, not checked yet. Write a function total_tokens(messages). It returns the sum of the tokens of the content of all messages in the list.
Task 3, not checked yet. Write a function fits(messages, model). It returns True when the total tokens of the messages are not more than the context limit of the model, and False when they are more. The limit is in MODELS.
Task 4, not checked yet. Make a list messages with a system message that has the content SYSTEM, and a user message that has the content comment. Call the model with max_tokens=5, and store the response in short. Make a variable was_cut. It is True when the stop reason of short is max_tokens.
Task 5, not checked yet. Write a function complete_answer(messages). It calls the model with max_tokens=10. When the stop reason is max_tokens, it calls the model again with max_tokens=200. It returns the text of the last answer.

Task 1 of 5

Make a variable comment_tokens with the number of tokens in comment. Use count_tokens.

All tasks