Lesson · 16 min

Conversations

Each call stands alone

A model service does not remember your last call. The model sees the messages of one call, and nothing else. When you send a question in a new call, the model does not know what you said before.

first = [{"role": "user", "content": "My name is Rosa."}]
second = [{"role": "user", "content": "What is my name?"}]

client.chat(first)
client.chat(second)   # The model does not know the name.

A chat app looks as if the model remembers. It does not. The app keeps the messages, and it sends them again with each call.

Send the conversation again

To continue a conversation, put the earlier messages in the list. An answer that the model gave before gets the role assistant.

history = [
    {"role": "user", "content": "My name is Rosa."},
    {"role": "assistant", "content": "Hello, Rosa."},
    {"role": "user", "content": "What is my name?"},
]
response = client.chat(history)

The model reads all messages, and it answers the last one. The last message of a call always has the role user.

Your program is the memory. After each call, it appends the answer to the list, so that the next call has it.

A conversation costs more with each turn

The list grows with each question and each answer, and each call sends the whole list. So the input tokens grow with each turn.

Turn	Messages in the call	You pay for
1	1	Question 1.
2	3	Question 1 and answer 1 again, and question 2.
3	5	All of turn 2 again, answer 2, and question 3.

A long conversation has two problems. Each call costs more than the one before. And after many turns, the messages do not fit the context window of the model.

Keep the list short

A program sets a limit of tokens for the history. When the history is longer, the program removes the oldest messages. It keeps the system message, because the instructions are necessary for each call.

The model then knows nothing of the messages that you removed. Some programs replace the old messages by a short summary, so that the main facts stay.

Some services want a user message directly after the system message. With such a service, remove a question together with its answer.

Your turn

For this lesson, the simulated model knows one more job. It reads a name in the messages, and it can repeat the name. Do the tasks in order.

Tasks

4 to do

Task 1, not checked yet. Call the model with one user message: My name is Rosa. Store the response in first. Then make a second call with one user message: What is my name? Store the text of its answer in forgot.
Task 2, not checked yet. Make a list history with three messages: the user message My name is Rosa., then an assistant message with the text of the answer in first, and then the user message What is my name? Call the model with history, and store the text of the answer in remembered.
Task 3, not checked yet. Write a function ask(history, question). It appends a user message with the question to history. It calls the model with history. It appends the answer to history as an assistant message. It returns the text of the answer.
Task 4, not checked yet. Write a function trim(history, limit). It returns a new list with the messages of history. While the messages have more than limit tokens, it removes the oldest message that is not a system message. It always keeps the last message. It does not change history.

Task 1 of 4

Call the model with one user message: My name is Rosa. Store the response in first. Then make a second call with one user message: What is my name? Store the text of its answer in forgot.

All tasks