Skip to main content

Streaming

Set "stream": true and the gateway relays server-sent events in the wire format of the surface you called — OpenAI-style chat.completion.chunk deltas or Anthropic-style message events. Your existing SDK's streaming helpers work unchanged.

OpenAI surface​

stream = client.chat.completions.create(
    model="gpt-5.1",
    stream=True,
    messages=[{"role": "user", "content": "Write a haiku about gateways"}],
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

What the gateway guarantees​

  • Failover happens before the first token. If the primary upstream is down, the request silently retries the fallback chain; once a stream has started it sticks to its upstream.
  • Usage arrives at the end. The final chunk carries the usage block; cost accounting lands in your dashboard seconds later.
  • Keep-alives. Long generations emit comment keep-alives so proxies don't kill the connection.

Handling errors mid-stream​

A stream that dies mid-generation ends with an error event (or a dropped connection). Treat partial output as disposable: retry the whole request — you are billed for what was generated, so prefer short max_tokens while iterating.

tip

Playground streams through this exact path — it's a live debugger for your streaming integration.