If you have built anything with a large language model, you have almost certainly used a chat completion API. It is the workhorse endpoint behind chatbots, assistants, agents, and most AI features shipping today: you send a list of messages, the model replies with the next message. The format is simple on the surface, but understanding it well — the message roles, the parameters, streaming, tools, and structured output — is the difference between an app that feels solid and one that fights you.
It helps to know one more thing up front: the same chat completion request can reach almost any model if you send it through the right layer. A tool like OrcaRouter is an OpenAI-compatible LLM gateway that accepts standard chat completion calls and routes them to 200+ models, so the interface you learn here works everywhere. This guide covers how the chat completion API works, its key features, best practices, and how to use one endpoint for every model.
What is a chat completion API?
A chat completion API takes a conversation — a list of messages — and returns the model’s next message. Each message has a role and content. The role is usually one of three: a system message that sets the model’s behavior and constraints, user messages that carry the human’s input, and assistant messages that hold the model’s previous replies. You send the whole conversation each time (the API is stateless), and the model generates the next assistant turn. This message-based shape is what distinguishes a chat completion API from older “text completion” APIs that simply continued a raw string.
How a chat completion request works
A request carries three things: the messages, the model name, and a handful of settings. The most common settings are the maximum number of output tokens (a cap on how long the reply can be) and whether to stream the response. You send the request over HTTPS with your API key, and you get back an assistant message plus usage information — how many tokens went in and came out, which is what you are billed on.
Because the conversation is sent in full each time, managing the message history is part of your job: you decide what to keep, what to summarize, and what to drop as a conversation grows. Keeping the stable parts of your prompt consistent also lets caching kick in, which lowers cost and latency on repeated context.
Streaming, tools, and structured output
Three features turn a basic chat completion API into a foundation for real products:
- Streaming — tokens arrive as they are generated instead of all at once, so a UI can show the reply as it types rather than after a long pause
- Tools (function calling) — the model can ask to call a function you define, letting it fetch data or take actions; you run the tool and feed the result back
- Structured output — you can constrain the reply to a JSON schema, so the model returns machine-readable data you can trust to parse
Support for these varies by model, which is one more reason to use an interface that lets you switch models freely without changing how you call them.
The chat completions standard — and why it matters
The chat completion format popularized by OpenAI has become the de facto standard. So many providers, SDKs, and tools now speak it that “OpenAI-compatible” is a feature you actively look for. The payoff is portability: code written against the standard chat completion API works across services with only a base-URL change, so you are never locked into one provider’s dialect.
One chat completion API for every model
The most useful application of the standard is to put it in front of a gateway. Instead of a chat completion endpoint that reaches a single model, a gateway’s endpoint reaches hundreds — so the same request format gives you access to models from many providers, with routing and failover built in. OrcaRouter works exactly this way: send a standard chat completion call to one OpenAI-compatible endpoint and it reaches 200+ models, with automatic failover and zero markup on tokens.
Best practices
A few habits keep chat completion apps reliable and cheap. Use a clear, stable system message and keep it consistent so caching works. Stream responses in any user-facing surface so the app feels fast. Set a sensible max-output limit to avoid runaway replies and surprise bills. Use structured output when you need to parse the result, rather than regexing free text. And route simple, high-volume calls to smaller models — the chat completion format is identical, so it costs you nothing to send cheap work to cheap models. Finally, handle errors and rate limits gracefully: retry transient failures with backoff, and fall back to another model when a provider is unavailable, so a single outage never breaks your feature. These habits cost little to add early and save real pain once you are at scale.
How to get started
- Create a free account and generate an API key
- Point your existing OpenAI SDK at the gateway’s base URL
- Send a chat completion request with a system message and your user’s input
- Add streaming, tools, or structured output as your product needs them
The bottom line
The chat completion API is the one interface every AI builder ends up knowing by heart — so learn it once, and make that knowledge portable. Written against the standard and pointed at a gateway, the same handful of calls carries you from your first prototype to a multi-model production app without ever being rewritten.
Want one chat completion endpoint for every model? Start free with OrcaRouter — standard chat completion calls to 200+ models, with automatic failover and zero markup.
FAQ
What does the temperature parameter actually do?
It scales how adventurous the model’s word choices are — low values make output focused and repeatable, higher ones more varied and creative. Note that some newer models fix or remove sampling parameters entirely, so don’t build product logic that depends on them.
Why did my conversation suddenly get worse after many turns?
You may be silently overflowing the context window: when history exceeds the model’s limit, your app has to truncate, and badly chosen truncation drops the instructions or facts the model needed. Summarize old turns instead of chopping them mid-conversation.
What’s the difference between stop reasons like “stop” and “length”?
Every response reports why generation ended: a natural finish, the max-token cap, or a tool call to run. Checking it matters — a “length” stop means the answer was cut off, and treating it as complete is a common silent bug.
