Lesson · 16 min

Choose a model

More than one model

A model service has a list of models. They are different in three ways.

	A large model	A small model
Quality	It gives more right answers for hard texts.	It is good for simple jobs.
Price	High.	Low.
Time	It answers more slowly.	It answers fast.

No model is the best for each job. The largest model is often not necessary, and the smallest model is sometimes not good enough. You cannot know this before you measure it.

Each call can name its model with the parameter model.

client.chat(messages, model="sim-large")

The simulated service has two models, sim-small and sim-large. The two models follow fixed rules, and the large model has more rules. So it reads a comment with more care. With real models, the difference comes from the size of the model. You measure it in the same way.

A test set

To measure a model, you need examples that have a known right answer. Such a list is a test set. A person reads each example and writes the right answer.

The file labelled.jsonl is a test set. It has the 20 comments of lesson 12, and a person gave each comment a label. Each line of the file is one JSON object with three keys. This is the line of the comment c06.

Key	Value
`id`	`c06`
`text`	`Easy to use, but the dock at Town Hall was full.`
`label`	`complaint`

This test set is small, so that the lesson runs fast. A real test set has 50 to 100 examples or more. It needs the hard cases too:

A comment with praise and a complaint.
A question that is also a complaint.
A comment that means the opposite of its words.

Accuracy

Run the model on each example, and compare its answer with the right answer. The accuracy is the part of the answers that are right, as a percentage. 12 right answers of 20 are 60 percent.

Read the wrong answers

The accuracy is one number. It does not tell you what the model cannot do. Read the examples that the model got wrong.

A group of wrong answers of the same kind shows a weak point of the model. Sometimes better instructions repair it. The next course, Prompt design, is about that.
Some wrong answers show a mistake in the test set. Correct the label.
For some examples, two people do not agree on the label. A model cannot do better than the people who made the test set.

The decision

Put the quality and the cost of each model next to each other. Then ask two questions.

What does a wrong answer cost? A wrong label on a comment is a small problem. A wrong answer to a customer about a payment is a large problem.
How many calls does the job make? A price that is 12 times higher is not important for 100 calls a day. It is important for a million calls.

Some programs use a small model for each item, and a large model only for the items where the small model is not sure.

Your turn

main.py reads the test set into the list labelled. Do the tasks in order.

Tasks

5 to do

Task 1, not checked yet. Write a function run(model). For each item in labelled, it calls the model with the system message SYSTEM, the text of the item as the user message, and the parameter model=model. It returns a list with one row for each item. A row has four keys: id, expected with the label of the item, got with the text of the answer, and cost with the result of cost(response["usage"], model).
Task 2, not checked yet. Write a function accuracy(rows). It returns the percentage of the rows in which got is equal to expected. Round the result to 1 decimal place.
Task 3, not checked yet. Make a list small with run("sim-small") and a list large with run("sim-large"). Make the variables accuracy_small and accuracy_large with the accuracy of each list.
Task 4, not checked yet. Make a variable cost_small with the sum of the costs in small, and a variable cost_large with the sum of the costs in large. Round each sum to 6 decimal places. Print the two accuracies and the two costs.
Task 5, not checked yet. Make a list wrong_small with the id of each row in small in which got is not equal to expected. Make a list wrong_large in the same way for large. Print the two lists.

Task 1 of 5

Write a function run(model). For each item in labelled, it calls the model with the system message SYSTEM, the text of the item as the user message, and the parameter model=model. It returns a list with one row for each item. A row has four keys: id, expected with the label of the item, got with the text of the answer, and cost with the result of cost(response["usage"], model).

All tasks