Streaming

Set stream: true to receive the answer as server-sent events while it is generated, instead of one JSON body at the end.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.fastrun.si/v1", api_key=os.environ["FASTRUN_API_KEY"])

stream = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.fastrun.si/v1", apiKey: process.env.FASTRUN_API_KEY });

const stream = await client.chat.completions.create({
  model: "google/gemma-4-31b-it",
  messages: [{ role: "user", content: "Write a haiku about GPUs." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

curl -N https://api.fastrun.si/v1/chat/completions \
  -H "Authorization: Bearer $FASTRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31b-it",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a haiku about GPUs."}]
  }'

Event format

Each event is a data: line with one chat.completion.chunk, and the stream ends with data: [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Silicon"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Token usage

To get the token counts of a stream, set stream_options.include_usage. The last chunk before data: [DONE] then carries usage and an empty choices list:

{
  "model": "google/gemma-4-31b-it",
  "stream": true,
  "stream_options": { "include_usage": true },
  "messages": [{ "role": "user", "content": "Hello!" }]
}

Your usage is counted either way; this option only decides whether you see the counts.

Keep-alive comments

When a stream has nothing to send for a few seconds, such as while a long prompt is read, fastrun sends a comment line so that proxies and clients don’t close the connection:

: keep-alive

Lines that start with : are comments in the server-sent events format. OpenAI SDKs and other SSE clients skip them; if you parse the stream yourself, skip them too.

Errors during a stream

If no GPU server has answered within 2 seconds, fastrun starts the response right away with status 200 and keeps it open with keep-alive comments until one does. If the request then fails, the status line has already been sent, so the error arrives as an event instead:

data: {"error":{"message":"no backend is available for this model","type":"upstream_error","code":503}}

OpenAI SDKs raise this as an API error. If you parse the stream yourself, check each event for an error field. Treat it like an HTTP error with the same code: see Errors and retries.

Stopping early

Closing the connection stops generation. Use it when the user cancels or you have what you need.

Get started

Models

Guides

Event format

Token usage

Keep-alive comments

Errors during a stream

Stopping early

​Event format

​Token usage

​Keep-alive comments

​Errors during a stream

​Stopping early

Event format

Token usage

Keep-alive comments

Errors during a stream

Stopping early