Many items

From one call to many

The lessons before this one made one call, or a few. A real job has many items. The file comments.jsonl has 20 comments of riders. A real system gets thousands each day.

Many calls bring three new problems.

• One call fails, and the program stops. The work on the other items is lost.
• After the run, you do not know which items have an answer.
• You do not know how many tokens the run used, and what it cost.

One item, one row

Write a function that does the complete work for one item. It returns one row in each case. When the call works, the row has the answer and the numbers of the call. When the call fails, the row has the error.

{"id": "c01", "label": "praise", "tokens": 67,
 "cost": 0.00003625}

{"id": "c04", "error": "ServerError"}

The function catches the errors of the service. So an error in one item does not stop the loop.

In this lesson, the function has the name process, and it has these rules.

• process(comment) gets a dictionary with the keys id and text.
• It calls the model with the system message STRICT, and the text of the comment as the user message.
• When the call works, it parses the answer with json.loads. It returns a dictionary with four keys: id, label, tokens and cost.
• label comes from the answer. tokens is the input tokens plus the output tokens. cost is the result of cost(response["usage"]).
• When the call raises a ModelError, it returns a dictionary with two keys: id and error. The value of error is the name of the class of the error.

The function cost(usage) of the simulation calculates the price of a call for the model sim-small. It does what your function call_cost of lesson 4 does.

To keep this lesson short, process uses the JSON of the answer and does not check it. The project of this course adds the checks of lesson 8.

A file with the results

Write each row to a file. The JSON Lines format is good for this. It has one JSON object on each line. Lesson 11 of the course Python for data explains it.

with open("results.jsonl", "w", encoding="utf-8") as file:
    for row in results:
        file.write(json.dumps(row) + "\n")

The file is the record of the run. You can read it again later, find the rows that have an error, and run only those items again. That costs less than a second run of all items.

A program that runs for a long time writes each row directly after its call. Then the rows are safe when the program stops in the middle.

The totals

Add up the numbers at the end of a run: the items that are done, the items that failed, the tokens, and the cost. A summary shows a problem early. Examples are a run with many failed items, and a cost that is much higher than the cost of the last run.

The time of a run

This lesson makes one call after the other. With 20,000 items and one second for each call, such a run takes more than 5 hours. Real programs send some calls at the same time, and they stay below the rate limit of the service. Some services also have a batch mode. You send one file with many calls, the answers come later, and the price is often lower.

Your turn

comments.jsonl has 20 comments. Do the tasks in order.