Lesson ยท 14 min
One change at a time
What the accuracy hides
A change in a prompt can repair some rows and break other rows. The accuracy shows only the difference. A prompt that repairs two rows and breaks one row has one more right answer, and the number goes up. You do not see the row that broke.
So compare the rows of the old version and the rows of the new version, one by one.
| Before | After | The row is |
|---|---|---|
| wrong | right | fixed |
| right | wrong | broken |
| right | right | not changed |
| wrong | wrong | not changed |
Read each broken row. Sometimes a broken row is a good price for five fixed rows. Often it shows that the new rule is too wide.
A rule that is too wide
Two wrong comments of lesson 3 have the word not: The ride was not good. and Good price, but the seat was not clean. A fast rule is: each comment that has the word not is a complaint.
That rule also changes this comment: I did not expect such a fast bike. Love it. It has the word not, and it is praise.
A better rule names the exact case. The word not changes the meaning of the praise word that comes after it: not good, not clean. In the comment about the fast bike, the word after not is expect, and the rule leaves that comment as it is.
One change at a time
When you change two things and the result is worse, you do not know which change did it. When the result is better, you do not know whether one of the two changes did harm that the other change hides.
Change one thing. Measure. Read the fixed rows and the broken rows. Then decide.
Rules work together
The comment Good price, but the seat was not clean. needs two rules. The rule about not makes clean a complaint. Then the comment has praise and a complaint, and the rule of lesson 3 makes it a complaint. A rule can depend on another rule. That is one more reason to measure each step.
Your turn
main.py has the prompt V3 of lesson 3 and its rows in rows_v3. The simulated model knows the two rules about not of this lesson by their key words, so write them as the tasks give them. Do the tasks in order.
Tasks
4 to do
Task 1, not checked yet. Write a function
compare(before, after). The two lists have rows for the same items in the same order. The function returns a dictionary with two keys.fixedis a list with theidof each row that was wrong inbeforeand is right inafter.brokenis a list with theidof each row that was right inbeforeand is wrong inafter.Task 2, not checked yet. Two of the wrong comments have the word
not. Try this rule:A comment that has the word not is a complaint.Make a promptV4_BROAD: the text ofV3, then one space, then the rule. Make a variablechange_broadwithcompare(rows_v3, run(V4_BROAD, dev)). Print it.Task 3, not checked yet. The rule is too wide: it broke the comment
c21. Use this rule in its place:The word not in front of a praise word makes a complaint.Make a promptV4: the text ofV3, then one space, then the new rule. Make a variablechangewithcompare(rows_v3, run(V4, dev)). Print it.Task 4, not checked yet. Make the variables
accuracy_broadandaccuracy_v4with the accuracy ofV4_BROADand ofV4ondev. Print the two numbers.
Task 1 of 4
Write a function compare(before, after). The two lists have rows for the same items in the same order. The function returns a dictionary with two keys. fixed is a list with the id of each row that was wrong in before and is right in after. broken is a list with the id of each row that was right in before and is wrong in after.