Lesson · 12 min

Temperature and steady answers

The same call, another answer

A function such as len gives the same result for the same input each time. A model does not. A model writes its answer one token after the other. For each step, it calculates how likely each possible token is, and then it picks one. So two calls with the same messages can give two different answers.

The parameter temperature controls the pick.

Temperature	What the model does	Use it for
`0`	It picks the most likely token each time. The same call gives the same answer, or almost the same answer.	Labels, data from a text, and each answer that a program reads.
Higher, such as `0.8`	It sometimes picks a token that is less likely. The answers vary.	Ideas, drafts, and texts where variety helps.

The simulated service accepts a temperature from 0 to 1. The range of a real service can be different, for example 0 to 2. Some models do not let you change the temperature.

What the simulated model changes

At a temperature above 0, the simulated model changes the form of its answer: complaint, Complaint, complaint. or Complaint. A real model can also change the content. At a high temperature, it can give another label for the same comment.

A real service picks at random, so you cannot repeat an answer. So that this lesson has the same result on each run, the simulated service has the parameter seed. The same seed gives the same pick.

client.chat(messages, temperature=0.8, seed=2)

Some real services have a seed too. They do not promise that the same seed always gives the same answer.

Why a program needs steady answers

Comparisons. "Complaint." == "complaint" is False. A program that compares the answer with a label fails on a small change of the form.
Tests. You cannot test a program when its correct output changes from run to run.
Errors. When a wrong answer comes one time in ten, the cause is hard to find.

Two steps for steady answers

Use temperature=0 for each answer that a program reads.
Normalize the answer before you compare it. Remove the spaces at both ends and the period at the end, and change the letters to lower case.

Do both. Also with temperature 0, a model can change the form of an answer now and then.

answer = " Complaint.\n"
answer.strip().removesuffix(".").lower()   # "complaint"

Your turn

main.py has the instructions and the messages. Do the tasks in order.

Tasks

5 to do

Task 1, not checked yet. Call the model five times with messages and temperature=0.8. Give the calls the seeds 1, 2, 3, 4 and 5 with the parameter seed. Make a list varied with the text of the five answers. Print it.
Task 2, not checked yet. Make a variable different with the number of different texts in varied.
Task 3, not checked yet. Call the model five times with messages and temperature=0. Make a list steady with the text of the five answers.
Task 4, not checked yet. Write a function normalize(answer). It removes the spaces at both ends of the text, then one period at the end, and it changes the letters to lower case. It returns the result.
Task 5, not checked yet. Write a function classify(comment). It calls the model with the system message SYSTEM, the comment as the user message, and temperature=0. It returns the normalized answer.

Task 1 of 5

Call the model five times with messages and temperature=0.8. Give the calls the seeds 1, 2, 3, 4 and 5 with the parameter seed. Make a list varied with the text of the five answers. Print it.

All tasks