Lesson · 14 min

Measure before you change

A test set for the prompt

Lesson 13 of the course Model APIs used a test set to compare two models. A test set is also the tool to compare two prompts. It is a list of examples, and a person wrote the right answer for each.

The file dev.jsonl has 24 comments of riders with a label. Each line is one JSON object.

Key	Value
`id`	`c06`
`text`	`Easy to use, but the dock at Town Hall was full.`
`label`	`complaint`

The name of this set is the dev set. "Dev" is short for development. You use it while you develop the prompt.

Why you measure first

Measure the prompt before you change it. The first number is your starting point. Without it, you do not know whether a change helped.

One answer tells you little. A change can repair the comment that you look at, and break three other comments that you do not look at. The test set shows all of them.

One number, and the rows behind it

The accuracy is the percentage of answers that are right. It is one number, and it is easy to compare. It does not tell you what is wrong.

So keep the rows. Each row has the label that a person gave and the label that the model gave. Then group the wrong rows by their kind.

Kind	A person said	The model said
`complaint -> praise`	complaint	praise
`question -> complaint`	question	complaint

A kind that has many rows usually has one cause. One change in the prompt can repair the whole group. Lesson 3 does that.

Your turn

main.py reads the dev set into the list dev. BASE is the prompt of lesson 1. The model is the simulated model of lesson 1, so the numbers are the same on each run. Do the tasks in order.

Tasks

5 to do

Task 1, not checked yet. Write a function run(prompt, items). Each item is a dictionary with id, text and label. For each item, the function calls the model with the prompt as the system message and the text as the user message. It returns a list with one row for each item. A row has three keys: id, expected with the label of the item, and got with the text of the answer.
Task 2, not checked yet. Write a function accuracy(rows). It returns the percentage of the rows in which got is equal to expected, rounded to 1 decimal place.
Task 3, not checked yet. Make a list rows_v1 with run(BASE, dev), and a variable accuracy_v1 with its accuracy. Print the accuracy.
Task 4, not checked yet. Make a list wrong_v1 with the id of each row in rows_v1 in which got is not equal to expected.
Task 5, not checked yet. Make a dictionary mistakes. Count the wrong rows by their kind. The key of a kind is the expected label, then ->, then the label that the model gave, for example question -> complaint. Print the dictionary.

Task 1 of 5

Write a function run(prompt, items). Each item is a dictionary with id, text and label. For each item, the function calls the model with the prompt as the system message and the text as the user message. It returns a list with one row for each item. A row has three keys: id, expected with the label of the item, and got with the text of the answer.