A second test set

A prompt that fits one test set too closely

In lessons 3 to 5, you read the wrong answers of the dev set, and you changed the prompt until they were right. The prompt V5 has 100 percent on the dev set.

That number is too good. You wrote the rules and the example for exactly these 24 comments. The prompt fits them. It is not certain that it fits the comments that come tomorrow. The name of this problem is overfitting.

A second test set

The answer is a second set of comments with labels. You do not look at it while you work on the prompt. Its usual names are the holdout set or the test set. The file holdout.jsonl has 24 other comments.

Set	You use it	You read its wrong answers
Dev set	After each change.	Yes. They tell you what to change.
Holdout set	Only to check a version that you want to use.	With care. Do not write a rule for one of its comments.

When you write rules for the comments of the holdout set, it becomes a second dev set. Then you have no set that tells you the truth, and you need new comments.

What the numbers tell you

Compare the two sets.

• The two numbers are close: the prompt works for comments that it did not see.
• The dev number is much higher: the prompt fits the dev set too closely.
• A change makes the dev number better and the holdout number worse: the change did harm. Take it out.

Why an example can do harm

The example of lesson 5 has the word walk, and its label is complaint. The simulated model learned that walk points to a complaint. That repaired one comment of the dev set. A rider who walks to a station and likes the service now gets the wrong label.

A real model also takes more from an example than you want. You cannot see that in the prompt. You see it only in a test on other data.

Your turn

main.py has the prompts V4 and V5, the two test sets, and the functions run, accuracy and compare. Do the tasks in order.