From the simulation to a real service

This course used a simulated model, so that each lesson runs in your browser and gives the same result each time. This page tells you what to expect when you call a real model service on your own computer. Read it, and then answer the questions.

What is the same

The main ideas of this course apply to each real service.

A call sends a list of messages with roles, and it gets one answer.
The answer has the text, the reason why the model stopped, and the token counts.
You pay for the input tokens and for the output tokens.
The model does not remember the last call. Your program sends the conversation again.
The answers vary. A program uses a low temperature, asks for JSON, and checks each answer.
A call can fail. Some errors are temporary, and a new try after a wait can work.

What is different

In this course	With a real service
The model is the file `sim_model.py`, and it runs in your browser.	The model runs on the computers of the service. A call goes through the internet. It takes between less than a second and many seconds.
The key is `test-key-lesson`.	You make an account and add a way to pay. Then you make a key in the settings of the account.
The client is `Client`, and the method is `chat`.	Each service has its own Python package. The client, the method and the parameters have other names.
The system message is the first message of the list.	With some services, the instructions are a separate parameter, and not a message.
The answer is a dictionary, and the text is under `content`.	The answer is an object, and the text is deeper inside it. With some services, it is in a list of choices. With other services, it is in a list of content blocks.
The stop reason is `end` or `max_tokens`.	The reasons have other names, and there are more of them. For example, a model can refuse to answer.
The model knows three jobs for comments of riders.	A real model can do many jobs. It can also be wrong and write as if it is sure.
A context window has 300 or 1,500 tokens.	A context window has 100,000 tokens or more.
The prices are made up.	The service has a price list, and the prices change.

The first steps on your computer

Choose a service, and open the page "Quickstart" in its documentation. The page has the exact code for Python.
Make an account. Set a spending limit, when the service has one.
Make a key, and put it in an environment variable, as in lesson 11.
Install the Python package of the service with pip. The quickstart page has the name of the package.
Send one short message. Print the complete answer, and find three things in it: the text, the stop reason, and the token counts.
Write one small function that calls the package and returns these three things in your own form. The other parts of your program call only this function.

What the package does

The package of a service sends a request through the internet, with the HTTPS protocol. The request has JSON text with the model and the messages. The key goes in a header of the request. The answer is JSON text too.

This example shows such a request with the modules that come with Python. The address, the names of the fields, and the header are examples. The documentation of your service has the correct ones. The code cannot run in a lesson, because a lesson has no access to the internet.

import json
import os
import urllib.request

messages = [
    {"role": "user", "content": "Classify this comment: The dock was full."},
]
body = {"model": "example-model", "max_tokens": 100, "messages": messages}
request = urllib.request.Request(
    "https://api.example.com/v1/chat",
    data=json.dumps(body).encode("utf-8"),
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer " + os.environ["MODEL_API_KEY"],
    },
)
with urllib.request.urlopen(request, timeout=30) as reply:
    answer = json.loads(reply.read())

When a call fails, the service answers with a status code. These are the codes of the errors of lesson 9.

Status code	Meaning	Error in this course
400	The call is wrong.	`BadRequestError`
401	The key is missing or wrong.	`AuthError`
429	Too many calls or tokens in a short time.	`RateLimitError`
500, 503	The service has a problem, or it has too much work.	`ServerError`

Rules for real calls

Private data. A call sends your text to another company. Send personal data of other people only when the terms of the service and the rules for the data allow it. Do not send passwords or the secrets of a business. Read how long the service keeps the text, and whether it uses the text to train models.
Money. Start with a small test. Calculate the cost of a large run before you start it, as in lesson 4.
Limits. A new account can make only a small number of calls in a minute. Write the program so that it waits and tries again.
Time. Set a time limit for each call. Without a limit, a program can wait for an answer for hours.
Model names. A service removes old models after some time. Keep the name of the model in a setting. Run your test set again when you change the model.
Wrong answers. A real model can write a wrong fact in a sure tone. Check each answer that is important.

The next course

The next course of the path AI developer is Prompt design. It shows how to write instructions that give a good answer each time, and how to test them.

Question 1 of 6

Which of these is the same with a real service as in this course?

A call sends a list of messages, and each message has a role.The Python package has the name sim_model.A key starts with test-key.The context window of a model has 300 tokens.

Question 2 of 6

You write a program that uses the Python package of a real service. Where do you put the code that calls the package?

In each place of the program that needs an answer of the model.In one small function. The other parts of the program call this function.In the file that has the key.In a comment, so that you can copy it.

Question 3 of 6

Your program will send the comments of customers to a model service. The comments have names and phone numbers. What do you do first?

Nothing. A model service does not store text.You send the comments with a higher temperature, so that the model does not learn them.You read the terms of the service and the rules for this data. You remove the personal data that the job does not need.You put the comments in the system message.

Question 4 of 6

A real service answers a call with the status code 429. What does it mean, and what does your program do?

The key is wrong. The program stops, and you correct the key.The messages have a wrong form. The program stops, and you correct the code.The program sent too many calls or too many tokens in a short time. It waits, and then it tries again.The model does not exist. The program uses another model.

Question 5 of 6

What do you do before the first large run with a real service? Choose all that apply.

Choose all answers that apply.

Set a spending limit in the account, when the service has one.Run a small test, and calculate the cost of the large run from its tokens.Write the key into the code, so that the run starts faster.Set max_tokens to a number that fits the longest answer that you expect.

Question 6 of 6

A service tells you that it will remove the model that your program uses. Which two things make the change to a new model safe? Choose all that apply.

Choose all answers that apply.

The name of the model is a setting, and not text in many places of the code.You have a test set, and you run it again with the new model.You use a higher temperature with the new model.You delete the result files of the old model.

Answer every question to check.