Lesson · 14 min

Rules for the hard cases

Read the wrong answers

Lesson 2 found two kinds of wrong answers for the prompt BASE. Read the comments of one kind, and look for what they have in common.

Kind	Two of its comments	What they have in common
`question -> complaint`	`Why is the station at Lighthouse closed? The app does not say why.` and `Where can I leave the bike when the dock at North Beach is full?`	The rider asks something, and the comment has a word such as `closed` or `full`.
`complaint -> praise`	`Easy to use, but the dock at Town Hall was full.` and `Great, the dock at Fish Market was full again.`	The comment has praise and a complaint.

The prompt does not say what to do in these cases. The model must choose, and it does not choose what you want.

A rule for a cause, and not for a comment

Write one rule for each cause. A rule is one short sentence in plain words.

A comment that asks something is a question.

Do not write a rule for one comment, such as The comment about Lighthouse is a question. That rule repairs one row of the test set, and it does not help with the comments of tomorrow.

Add one rule, and measure

Add the rule to the end of the prompt, and run the test set again. Then you know what this rule did. Lesson 4 shows why one change at a time is important.

Rules cost tokens

The prompt goes to the model with each call. So each rule adds its tokens to the cost of each call. Keep a rule when it repairs more than it costs. Lesson 9 calculates the cost.

Rules and the simulated model

The simulated model finds a rule by its key words. So write the rule sentences as the tasks give them. A real model reads the meaning of a sentence, and it understands other words for the same rule. For a real model too, a short and clear sentence works better than a long one.

Your turn

main.py has the functions run and accuracy of lesson 2. Do the tasks in order.

Tasks

4 to do

Task 1, not checked yet. Two questions got the label complaint. Make a variable RULE_QUESTION with this rule: A comment that asks something is a question. Make a prompt V2: the text of BASE, then one space, then the rule. Make a variable accuracy_v2 with the accuracy of V2 on dev.
Task 2, not checked yet. Six complaints got the label praise. Some of them have praise and a complaint in one comment. Make a variable RULE_MIXED with this rule: A comment that has praise and a complaint is a complaint. Make a prompt V3: the text of V2, then one space, then the rule. Make a variable accuracy_v3 with the accuracy of V3 on dev.
Task 3, not checked yet. Make a list still_wrong with the id of each row that is wrong with V3. Print the text of these comments, so that you can read them.
Task 4, not checked yet. A longer prompt costs more tokens with each call. Make a variable tokens_added with the tokens of V3 minus the tokens of BASE. Use count_tokens.

Task 1 of 4

Two questions got the label complaint. Make a variable RULE_QUESTION with this rule: A comment that asks something is a question. Make a prompt V2: the text of BASE, then one space, then the rule. Make a variable accuracy_v2 with the accuracy of V2 on dev.

All tasks