VEL ORENDISVel Orendis
The Observatory / Working with LLMsQuest: The Growing Bill
0 StandingLV 10 XP

The Price of a Conversation

Watch one conversation go out, call by call.

Tidewater's assistant costs more with every question. Find out where the tokens go.

Tidewater Bootcamp's assistant runs on a model it calls over an API. Each call sends a list of messages, and each message has a role: the system message sets the rules, user messages are the student's, and assistant messages are the model's earlier replies.1

The model keeps nothing between calls. To carry on a conversation, the app sends the history again, every time.

The provider counts every message in tokens, pieces of words, and charges per million: once for what goes in, and at a higher rate for what comes out.1

Watch one conversation: six questions, six calls.

References

  1. 1. OpenAI Cookbook, How to format inputs to ChatGPT models: messages and their roles; How to count tokens with tiktoken: encodings differ by model, and counting a chat request. OpenAI, MIT, commit 5bd6cb4.
  2. 2. Claude Cookbooks, Session memory compaction: long conversations outgrow the request; Sampling past max_tokens: the output ceiling. Anthropic, MIT, commit a162e01.

Numbered ideas come from these sources, rewritten in our own words. Everything else, including the simulation and its numbers, was written for Vel Orendis.

Input tokens, call by call
  • System message
  • Earlier turns
  • New question
  • Right
  • Wrong
What call 1 sends

POST https://api.provider-a.example/v1/chat/completions

model
a-medium
max_tokens
400
messages
2
  1. systemYou are the assistant for Tidewater Bootcamp. Answer students' questions from the FAQ below. If the FAQ does not answer a question, say you do not know. Answer in a short paragraph. FAQ: Start dates, Course length, Hours, Tuition, Instalments, Equipment, Moving cohorts, Refunds, Location, Certificate, Careers, Missed sessions.377
  2. usernewHi, I have a place on the part-time evening track. When does it start?23

Input: 403 tokens, counting 3 for each message’s role and 3 to start the reply.

The reply, 27 output tokens, finish_reason stop

The part-time evening track starts on 2 February. The evening track meets online, so you can join from home.

Call 1 sends two messages: the system message, with the rules and Tidewater's 12 most asked questions, and the student's first question.

When you have watched the six calls go out, open the request.