Your first call to a model

A model as a service

A language model is a program that reads text and writes text. Many teams do not run the model on their own computers. A company runs it, and your program sends a request to it through the internet. The company calls this a model API. An API is a way for one program to use another program.

Each request costs money. The price depends on the amount of text that goes in and comes out.

The model in this course

A lesson of this site runs in your browser, and the code has no access to the internet. So this course uses a simulated model. It is the file sim_model.py in the tab next to main.py. It follows fixed rules. It knows three jobs for the comments of bike riders: it can classify a comment, list the stations in it, and summarize it.

A real model can do much more, and its answers change more from call to call. The other things are the same as with a real service: the call, the parts of the answer, the tokens, the cost, and the errors. You can read sim_model.py to see how the simulation works.

Messages

A request holds a list of messages. Each message is a dictionary with two keys.

messages = [
    {
        "role": "user",
        "content": "Summarize this: The dock was full.",
    },
]

• role says who wrote the message. The role user is for text that comes from you or from your program.
• content is the text of the message.

The call

from sim_model import Client

client = Client(api_key="test-key-lesson")
response = client.chat(messages)

A Client object holds the settings of the connection, such as the key. Its method chat sends the messages, waits for the model, and returns a dictionary.

Key	Value
content	The text that the model wrote.
stop_reason	The reason why the model stopped. "end" means that the answer is complete.
usage	The tokens of the call, as a dictionary with input_tokens and output_tokens.
model	The name of the model that answered.
id	A name for this call. It helps when you search a log.
latency_ms	The time that the answer took, in milliseconds.

A token is a small piece of text, about one word. Lesson 3 explains tokens.

Your turn

main.py already makes the client. Do the tasks in order.