Project · 45 min

Project: a report on rider comments

The job

The bike share service gets comments from its riders. The file week_comments.jsonl has the 24 comments of one week. The team that runs the service has three questions.

How many comments are praise, complaints and questions?
Which stations get the most complaints?
What does it cost to let a model read the comments?

You write the program that answers them. It uses each part of this course: messages, JSON answers, checks, new tries after an error, a result file, and the cost.

The model in this project is still the simulated model.

One call for three results

The instructions in PROMPT ask for three results in one JSON object. The model writes the object on one line. It is on five lines here, so that you can read it.

{
  "label": "complaint",
  "stations": ["Harbor Gate"],
  "summary": "No free dock at Harbor Gate at 8 ..."
}

One call for three results costs less than three calls, because the comment and most of the instructions go to the model one time only.

MODEL is sim-large. Lesson 13 showed that this model gives more right labels.

What `main.py` already has

extract_json(text) of lesson 7. It finds the JSON object in an answer, or it returns None.
chat_with_retry(messages, max_attempts) of lesson 10. It tries a call again after a temporary error, and it raises the error when the last attempt fails.

The rows

The function analyze returns one row for one comment. Each row has the id, the tokens and the cost. A good row also has the three results. A row that failed has the key error.

{"id": "w01", "tokens": 111, "cost": 0.000675,
 "label": "complaint", "stations": ["Harbor Gate"],
 "summary": "No free dock at Harbor Gate at 8 ..."}

{"id": "w03", "tokens": 94, "cost": 0.00048,
 "error": "InvalidAnswer"}

{"id": "w06", "tokens": 0, "cost": 0.0,
 "error": "ModelTimeout"}

A row with the error InvalidAnswer has tokens and a cost. The model wrote an answer, and you pay for it, also when you cannot use it. A call that failed with an error of the service has no answer, so its tokens and its cost are 0.

The function analyze

analyze(comment) does the work for one comment in five steps.

It makes the row {"id": comment["id"], "tokens": 0, "cost": 0.0}.
It calls chat_with_retry with two messages and 3 attempts. The messages are the system message PROMPT, and the text of the comment as the user message.
When the call raises a ModelError, it adds the key error to the row, with the name of the class of the error. It returns the row.
When the call works, it sets tokens to the input tokens plus the output tokens. It sets cost to cost(response["usage"], MODEL).
It reads the answer with extract_json, and it tests the data with is_valid. When the data is not valid, it adds the key error with the text InvalidAnswer. When the data is valid, it adds the keys label, stations and summary from the data. It returns the row.

Two passes

The first pass runs analyze for each comment. Some rows fail. The function second_pass(results, comments) runs analyze again for those rows only. It has these rules.

The two lists have the same order. The first row belongs to the first comment. zip(results, comments) gives these pairs to a for loop.
The function returns a new list. It does not change the old list or the old rows.
A row that has no error goes into the new list as it is.
For a row that has an error, the function calls analyze for the comment of the row. It adds the tokens and the cost of the old row to the new row. Then the new row goes into the new list.

The new row has the tokens and the cost of the two tries. So the report shows the full cost of each comment.

Your turn

Do the tasks in order. Run the code after each task.

Tasks

7 to do

Task 1, not checked yet. Read the file week_comments.jsonl. Make a list comments with one dictionary for each line.
Task 2, not checked yet. Write a function is_valid(data). It returns True when all of this is true: data is a dictionary, its label is one of LABELS, its stations is a list, and its summary is text that is not empty. In each other case, it returns False.
Task 3, not checked yet. Write a function analyze(comment). It returns one row for one comment. The section "The function analyze" of the text has its five steps.
Task 4, not checked yet. Add the line plan(["ok", "server_error", "ok", "bad_json", "ok", "ok", "timeout", "timeout", "timeout"]). It makes some of the next calls fail. After that line, make a list results with the result of analyze for each comment. Make a list failed with the id of each row in results that has the key error. Print failed.
Task 5, not checked yet. Write a function second_pass(results, comments). It returns a new list of rows, and it calls analyze again for each row that has an error. The section "Two passes" of the text has its rules. Then make a list final with second_pass(results, comments).
Task 6, not checked yet. Write the file report.jsonl. It has one line for each row of final, and you make the line with json.dumps.
Task 7, not checked yet. Make a dictionary summary from the rows of final. It has six keys. done is the number of rows that have no error. failed is a list with the id of each row that has an error. labels has the number of rows for each label. complaints_by_station has, for each station, the number of complaints that name it. tokens is the sum of the tokens of all rows. cost is the sum of the costs of all rows, rounded to 6 decimal places. Then make a variable worst_station with the station that has the most complaints. Print summary and worst_station.

Task 1 of 7

Read the file week_comments.jsonl. Make a list comments with one dictionary for each line.

All tasks