Streaming

Pass stream: true in any chat-completions or completions request. The response is text/event-stream, OpenAI-compatible.

Chunk format

Each line is data: followed by a JSON chunk, separated by blank lines.

data: {"id":"...","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"...","choices":[{"delta":{"content":" world"},"index":0}]}

data: [DONE]

Cancellation

When the client disconnects, we abort the upstream connection and charge only for the tokens already generated. The llm_requests row is logged with status = cancelled.

Token usage

We honour OpenAI's stream_options: { "include_usage": true } when supplied. The final non-DONE chunk then includes usage.

Caveats

• Streaming over a corporate proxy that buffers Server-Sent Events can break the flush boundary — disable buffering or use raw HTTP/2.
• If our gateway crashes mid-stream, your reservation stays locked for up to 5 minutes until the sweeper releases it.