Lesson · 14 min

The cost of a prompt

You pay for the prompt with each call

The system message goes to the model with each call. A prompt with three rules and an example has more tokens than a prompt with no rule. You pay for these tokens each time.

Lesson 4 of the course Model APIs calculated the price of a call from its tokens. The function cost(usage, model) of the simulation does that calculation.

For one call, the difference is small. So calculate the cost of many calls. This lesson uses 100,000 comments.

Accuracy has a price

Put the accuracy and the cost of each version in one table. Then you see what each step costs.

A rule that repairs many rows for a few tokens is a good trade.
An example that repairs one row and adds many tokens is a worse trade.
A part that repairs nothing costs money for no result. Take it out.

The table of this lesson uses the dev set. For a real decision, use the numbers of the holdout set too, as in lesson 6.

A better prompt, or a larger model

Lesson 13 of the course Model APIs compared a small model and a large model. The large model gave more right answers, and it cost 12 times more for the same tokens.

A better prompt is a second way to more right answers. It makes each call longer, so it costs more too. The tasks of this lesson show which way costs less for this job.

Try the prompt first. It is the cheaper change, and you learn what the job needs. Use a larger model when a good prompt on the small model is not enough.

Two more facts about long prompts

A longer prompt takes the model more time to read. For a program that answers a person, the wait is important.
Some services charge less for the start of a prompt when it is the same in each call. With such a service, put the fixed parts first and the comment last. The prompts of this course have this order.

Your turn

main.py has the prompts V1, V4 and V5, and the dev set. Do the tasks in order.

Tasks

4 to do

Task 1, not checked yet. Write a function run_cost(prompt, items, model). It works as run of lesson 2, with two changes. It calls the model with the parameter model=model. And each row has two more keys: tokens with the input tokens of the call, and cost with the result of cost(response["usage"], model).
Task 2, not checked yet. Write a function summary(rows). It returns a dictionary with three keys. accuracy is the percentage of right rows, rounded to 1 decimal place. tokens is the mean of the key tokens, rounded to 1 decimal place. cost is the cost of 100,000 comments in dollars: the mean of the key cost, times 100,000, rounded to 2 decimal places.
Task 3, not checked yet. Make a dictionary table with three keys, v1, v4 and v5. Each value is the summary of that prompt on dev with the model sim-small. Print the three summaries.
Task 4, not checked yet. Compare with the large model. Make a variable large with the summary of V1 on dev with the model sim-large. Make a variable times_cheaper: the cost of large divided by the cost of v4 in table, rounded to 1 decimal place. Print the two values.

Task 1 of 4

Write a function run_cost(prompt, items, model). It works as run of lesson 2, with two changes. It calls the model with the parameter model=model. And each row has two more keys: tokens with the input tokens of the call, and cost with the result of cost(response["usage"], model).

All tasks