Lesson · 14 min

Examples in a prompt

A case that no rule describes

One comment of the dev set is still wrong with the prompt V4: Thanks for the free walk home. The words are friendly. The meaning is a complaint: the rider found no bike, and walked. A rule for this case is hard to write in one sentence.

When you cannot say a rule, show an example. An example in a prompt is an input together with the answer that you want.

Examples:
Comment: Thanks for nothing. I had to walk.
Label: complaint

The model reads the example, and it uses what it sees there for the comments that are similar.

The form of the examples

Put the examples after the rules, under a line such as Examples:.
Give each example the same form: the same words in front of the input and in front of the answer.
Write the answer in the exact form that you want from the model. For this prompt, that is one word.
Use a comment that is not in the test set. An example that is a copy of a test comment tells you nothing about other comments.

A line break in a Python text is "\n". The method join puts a text between the items of a list.

lines = ["Examples:", "Comment: Great ride.", "Label: praise"]
text = "\n".join(lines)

How many examples

Each example is in the prompt of each call, so you pay for its tokens each time. Start with examples for the cases that the rules do not repair. Two or three good examples are often enough. Measure each one.

An example can also do harm. The model can learn more from it than you want. Lesson 6 shows such a case, and how you find it.

Examples and the simulated model

The simulated model learns the new words of an example for the label of the example. From the example of this lesson, it learns that nothing and walk point to a complaint. A real model learns in a way that is harder to predict. That is the reason why you measure.

Your turn

main.py has the prompt V4 of lesson 4, and the functions run and accuracy. Do the tasks in order.

Tasks

4 to do

Task 1, not checked yet. Write a function example_block(pairs). pairs is a list, and each item has a comment and a label. The function returns a text. Its first line is Examples:. Then each pair adds two lines: Comment: with the comment, and Label: with the label. The lines have a line break between them, and the text has no line break at its end.
Task 2, not checked yet. Make a variable EXAMPLES with example_block for one pair: the comment Thanks for nothing. I had to walk. and the label complaint. Make a prompt V5: the text of V4, then a line break, then EXAMPLES.
Task 3, not checked yet. Make the lists wrong_v4 and wrong_v5 with the id of each wrong row of V4 and of V5 on dev. Make a variable accuracy_v5 with the accuracy of V5. Print the three values.
Task 4, not checked yet. Examples cost tokens. Make the variables tokens_v4 and tokens_v5 with the tokens of each prompt. Use count_tokens. Print the two numbers.

Task 1 of 4

Write a function example_block(pairs). pairs is a list, and each item has a comment and a label. The function returns a text. Its first line is Examples:. Then each pair adds two lines: Comment: with the comment, and Label: with the label. The lines have a line break between them, and the text has no line break at its end.

All tasks