Project · 45 min

Project: a prompt inside a budget

The job

The bike share service wants to give a label to each comment of its riders. The team has three demands for the prompt.

Demand	Limit
Accuracy on the dev set	90 or more
Accuracy on the holdout set	90 or more
Input tokens for a call on the dev set, as a mean	70 or less

The team also wants to know what the small model with your prompt costs, and what the large model with the short prompt costs.

No single answer is given to you. You test combinations of parts, you read the numbers, and you choose. The check of task 4 measures your prompt on the two sets. It accepts each prompt that reaches the three limits.

The model in this project is still the simulated model.

The parts

main.py has the parts that you can use.

BASE is the task and the form of the answer. Each prompt starts with it.
RULES has four rule sentences of this course, each under a short name.
EXAMPLES has one example under the name walk.

A combination is a list of rule names and a list of example names, such as ["question", "mixed"] and [].

The method

Build a prompt from a combination.
Measure it on the two sets.
Write the result into a row of the list tried.
Read the rows, and choose the combination that reaches the three limits.

A part can help on one set and do harm on the other set. A part can also cost more tokens than the limit allows. The rows of tried show both.

The functions

build_prompt(rule_names, example_names) builds the text, as in lesson 8. It gets names, and it takes the texts from RULES and EXAMPLES. An example in EXAMPLES is a pair: the comment, and then the label.

evaluate(prompt, items) measures one prompt on one set. It returns a dictionary such as this one.

{"accuracy": 87.5, "tokens": 52.9,
 "wrong": ["c11", "c13", "c20"]}

Your turn

Do the tasks in order. Run the code after each task.

Tasks

6 to do

Task 1, not checked yet. Write a function build_prompt(rule_names, example_names). The two lists have keys of RULES and keys of EXAMPLES. The function returns the text of BASE, then each rule with one space in front of it. When example_names is not empty, the text then gets a line break, the line Examples:, and the two lines Comment: ... and Label: ... for each example, as in lesson 5.
Task 2, not checked yet. Write a function evaluate(prompt, items). It calls the model for each item, as run of lesson 2. It returns a dictionary with three keys. accuracy is the percentage of right answers, rounded to 1 decimal place. tokens is the mean of the input tokens of the calls, rounded to 1 decimal place. wrong is a list with the id of each item that got a wrong answer.
Task 3, not checked yet. Test at least three combinations of parts. Make a list tried with one row for each combination. A row has five keys: rules and examples with the two lists of names, dev and holdout with the accuracy of the prompt on each set, and tokens with the mean input tokens on dev. Print each row.
Task 4, not checked yet. Choose the combination for the team. It needs an accuracy of 90 or more on dev, an accuracy of 90 or more on holdout, and 70 input tokens or less for a call on dev. Make a dictionary CHOICE with the keys rules and examples for your combination, and a variable PROMPT with build_prompt(CHOICE["rules"], CHOICE["examples"]).
Task 5, not checked yet. Make a dictionary report with two keys, dev and holdout. Each value is the result of evaluate for PROMPT on that set. Print it.
Task 6, not checked yet. Compare the cost with the large model. Make a variable cost_small: the cost of 100,000 comments with PROMPT and the model sim-small, from the calls for dev. Make a variable cost_large in the same way for the prompt BASE and the model sim-large. Round each to 2 decimal places. Make a variable saving with cost_large minus cost_small, rounded to 2 decimal places. Print the three numbers.

Task 1 of 6

Write a function build_prompt(rule_names, example_names). The two lists have keys of RULES and keys of EXAMPLES. The function returns the text of BASE, then each rule with one space in front of it. When example_names is not empty, the text then gets a line break, the line Examples:, and the two lines Comment: ... and Label: ... for each example, as in lesson 5.

All tasks