Model APIs cheat sheet

from sim_model import Client

client = Client(api_key="test-key-lesson")
text = "Summarize this: The dock was full."
messages = [{"role": "user", "content": text}]

response = client.chat(messages)
print(response["content"])

RULES = "Summarize the text in 8 words."
comment = "The dock at Town Hall was full."

messages = [
    {"role": "system", "content": RULES},
    {"role": "user", "content": comment},
]

Say what you want: the labels that you allow, and the form of the answer.

from sim_model import MODELS, count_tokens

count_tokens("The dock was full.")  # 5
MODELS["sim-small"]["context"]      # 300
MODELS["sim-large"]["context"]      # 1500

response = client.chat(messages, max_tokens=5)
response["stop_reason"]  # "max_tokens"

• A model reads and writes tokens. You pay for each token, and each call has a limit.
• The context is the limit of the input. Messages above it give a BadRequestError.
• max_tokens is the limit of the output. The stop reason "max_tokens" means that the service cut the answer.

from sim_model import MODELS, cost

price = MODELS["sim-small"]
dollars = (
    2000 * price["input_price"]    # input tokens
    + 500 * price["output_price"]  # output tokens
) / 1_000_000                      # 0.001

usage = {"input_tokens": 2000, "output_tokens": 500}
cost(usage)                     # 0.001
cost(usage, model="sim-large")  # 0.012

A price is for 1,000,000 tokens. Output tokens cost more than input tokens. The prices in this course are made up.

To lower the cost: use the smallest model that does the job well, keep the instructions short, ask for a short answer, set max_tokens, and do not ask twice for the same text.

history = [
    {"role": "user", "content": "My name is Rosa."},
    {"role": "assistant", "content": "Hello."},
    {"role": "user", "content": "What is my name?"},
]
response = client.chat(history)
answer = response["content"]
history.append(
    {"role": "assistant", "content": answer}
)

• The model does not remember a call. Your program sends the conversation again each time.
• Each turn sends all earlier turns, so each turn costs more than the turn before.
• Keep the list short, for example with only the last turns.

# Steady: for each answer that a program reads
client.chat(messages, temperature=0)
# With variety: for ideas and drafts
client.chat(messages, temperature=0.8)

# Normalize an answer before you compare it
answer = " Complaint.\n"
answer.strip().removesuffix(".").lower()
# 'complaint'

import json

STRICT = (
    "Classify the comment as one of: praise, "
    "complaint, question, other. Answer with only "
    'JSON. Use the keys "label" and "reason".'
)

text = '{"label": "complaint", "reason": "Full."}'
data = json.loads(text)
data["label"]  # 'complaint'

# Cut away the text around the JSON
text = 'Here is the JSON:\n{"label": "praise"}'
start = text.find("{")
end = text.rfind("}")
json.loads(text[start:end + 1])
# {'label': 'praise'}

Strict instructions name three things: only JSON, the keys, and the values that you allow. json.loads raises a ValueError for text that is not JSON.

LABELS = [
    "praise", "complaint", "question", "other",
]

def is_good(data):
    if not isinstance(data, dict):
        return False
    if "label" not in data:
        return False
    return data["label"] in LABELS

data = None
for attempt in range(3):
    response = client.chat(messages)
    try:
        answer = json.loads(response["content"])
    except ValueError:
        continue  # not JSON: ask again
    if is_good(answer):
        data = answer
        break

Set a limit for the tries. When no try works, use a default value, give the item to a person, or stop with an error. Do not go on with an answer that failed the check.

from sim_model import ModelError, RateLimitError

try:
    response = client.chat(messages)
except RateLimitError as error:
    print("Wait for", error.retry_after, "seconds.")
except ModelError as error:
    print(type(error).__name__, str(error))

ModelError is the parent class of the five errors. Put a special class before its parent class.

from sim_model import ServerError, wait

def backoff(attempt):
    # 1, 2, 4, 8, 16, and then 30 seconds
    return min(2 ** attempt, 30)

def chat_with_retry(messages, max_attempts):
    for attempt in range(max_attempts):
        try:
            return client.chat(messages)
        except ServerError:
            if attempt == max_attempts - 1:
                raise
            wait(backoff(attempt))

• The wait doubles after each failure. This is exponential backoff.
• For a RateLimitError, wait for error.retry_after seconds.
• Stop after some attempts, and raise the last error.
• Try again only for an error that a new try can fix.

import os

key = os.environ.get("MODEL_API_KEY", "")
if key == "":
    raise RuntimeError("Set MODEL_API_KEY.")

model = os.environ.get("MODEL_NAME", "sim-small")

• A key is a secret. Keep it out of the code, of messages, of the output, and of a web page.
• To show which key is in use, show only its last 4 characters.
• When a key got out, delete it in the account and make a new key.
• Set a spending limit, and use one key for each program.
• Do not type a real key into a lesson.

def process(comment):
    text = comment["text"]
    messages = [
        {"role": "system", "content": STRICT},
        {"role": "user", "content": text},
    ]
    try:
        response = client.chat(messages)
    except ModelError as error:
        name = type(error).__name__
        return {"id": comment["id"], "error": name}
    label = json.loads(response["content"])["label"]
    return {"id": comment["id"], "label": label}

results = [process(comment) for comment in comments]

with open("results.jsonl", "w",
          encoding="utf-8") as file:
    for row in results:
        file.write(json.dumps(row) + "\n")

One item gives one row: a result, or the name of the error. Then one failed call does not stop the run, and you know which items have no answer. Add the tokens and the cost of each row to see the totals of the run.

response = client.chat(messages, model="sim-large")

# Accuracy on a test set, in percent
right = 0
for example in examples:
    answer = classify(example["text"])
    if answer == example["label"]:
        right += 1
accuracy = 100 * right / len(examples)

• A test set is a list of examples with the right answer for each one.
• Read the wrong answers. A group of the same kind shows a weak point of the model.
• Choose with two questions: what does a wrong answer cost, and how many calls does the job make?

• Each service has its own package. The names of the client, the method and the parameters are different.
• Put the call to the package in one small function of your own.
• Send private data only when the rules for the data allow it.
• Calculate the cost of a large run before you start it.
• Set a time limit for each call.
• Keep the name of the model in a setting, and run your test set again when you change the model.