Lesson · 16 min

Check the answer, and try again

JSON that parses can still be wrong

json.loads checks one thing: the text is JSON. Your program needs more than that. An answer of a model can fail in three ways.

Problem	Example
The answer is not JSON, or it is cut.	`{"label": "complaint", "reas`
A key is missing.	`{"label": "complaint"}`
A value is not one that you allow.	`{"label": "unclear", "reason": "..."}`

Lesson 8 of the course Python for data showed that real files have mistakes, and that your code must find them. The answer of a model is data of the same kind. A model follows your instructions most of the time, and not each time.

Check the answer

Write down what a good answer is, and test each point in code.

The type: a dictionary, and not a list or a text.
The keys: each key that the program reads.
The values: a label from your list, a number in a range, a text that is not empty.

LABELS = ["praise", "complaint", "question", "other"]

data = {"label": "unclear", "reason": "The dock was full."}
data["label"] in LABELS   # False

Test the type first. "label" in data fails with an error when data is None.

The function isinstance(value, kind) tests the type of a value. The type of a dictionary is dict, the type of a list is list, and the type of a text is str.

isinstance({"label": "praise"}, dict)   # True
isinstance(None, dict)                  # False
isinstance("praise", str)               # True

Ask again

After a bad answer, a program can ask again. A new call often gives a good answer when the cause was a cut answer or a rare mistake in the form. When the model repeats its mistake, add the bad answer and a correction to the messages of the next call, such as Your answer was not valid JSON. Answer with only JSON.

Set a limit for the tries. Each try costs tokens, and each try takes time. Two or three tries are usual.

When no try works

Decide what the program does when it has no good answer after the last try. These are the usual choices.

Use a default value, such as the label unknown.
Put the item in a list for a person to check.
Stop with an error.

Do not continue with an answer that failed the check.

Bad answers on demand

A real model gives a bad answer now and then, and you cannot tell when. To test your code, the simulated service gives bad answers when you tell it to. The function plan takes a list with one step for each of the next calls.

from sim_model import plan

plan(["bad_json", "wrong_label", "ok"])

Step	What the call does
`"bad_json"`	The answer is cut in the middle of the JSON.
`"wrong_label"`	The answer has the label `unclear`.
`"ok"`	The answer is good.

After the last step of the plan, each call is good again.

Your turn

main.py has the function extract_json of lesson 7, and a function ask_once that makes one call. Do the tasks in order.

Tasks

4 to do

Task 1, not checked yet. Write a function is_valid(data). It returns True when all of this is true: data is a dictionary, it has the keys label and reason, the label is one of LABELS, and the reason is text that is not empty. In each other case, it returns False.
Task 2, not checked yet. Write a function ask_valid(comment, tries). It calls ask_once(comment) up to tries times. It returns the first data that is_valid accepts, and it makes no call after that. When no try gives valid data, it returns None.
Task 3, not checked yet. Add the line plan(["bad_json", "wrong_label", "ok"]). It tells the simulated service to give two bad answers and then a good answer. After that line, call ask_valid("The dock was full.", 3), and store the result in answer. Print it.
Task 4, not checked yet. Write a function safe_label(comment). It calls ask_valid with 2 tries. It returns the label of the data. When ask_valid gives None, it returns the text unknown.

Task 1 of 4

Write a function is_valid(data). It returns True when all of this is true: data is a dictionary, it has the keys label and reason, the label is one of LABELS, and the reason is text that is not empty. In each other case, it returns False.

All tasks