Errors and retries

Errors use the OpenAI format, with an HTTP status and a JSON body:

{
  "error": {
    "message": "all backends for this model are at capacity",
    "type": "rate_limit_error",
    "code": 429
  }
}

Status	`type`	Meaning	Retry?
400	`invalid_request_error`	The body is not a JSON object, or `model` is missing.	No. Fix the request.
400	(from the model)	The model rejected the request, for example a prompt longer than the context window.	No. Fix the request.
401	`authentication_error`	The key is missing, unknown or revoked, or the account is not approved yet.	No. See Authentication.
429	`rate_limit_error`	Every GPU serving the model is busy.	Yes, after `Retry-After`.
502	`upstream_error`	The GPU servers failed to answer.	Yes, with backoff.
503	`unavailable`	No GPU server for the model is up right now, or keys cannot be checked for a moment.	Yes, after `Retry-After`.

Before answering with 429, 502 or 503, fastrun has already tried the other GPU servers that serve the model. When these errors persist, wait longer between retries rather than retrying faster.

Retrying

Retry 429, 502 and 503 with exponential backoff, and wait at least as long as the Retry-After header says. OpenAI SDKs do this for you: they retry these statuses twice by default and follow Retry-After. You can raise the count:

client = OpenAI(
    base_url="https://api.fastrun.si/v1",
    api_key=os.environ["FASTRUN_API_KEY"],
    max_retries=5,
)

const client = new OpenAI({
  baseURL: "https://api.fastrun.si/v1",
  apiKey: process.env.FASTRUN_API_KEY,
  maxRetries: 5,
});

Don’t retry 400 and 401: the same request fails the same way.

Errors in a stream

A streamed response can start with status 200 before the request is sure to succeed. A failure after that point arrives as an event with an error field. See Streaming.

Timeouts

A streamed request can run for up to 60 minutes. A request without streaming sends nothing until the whole answer is ready and must be complete within 10 minutes, so set your client’s timeout to cover the longest answer you expect. For long answers, use streaming, which sends tokens and keep-alive comments as it goes.

Get started

Models

Guides

Errors and retries

Retrying

Errors in a stream

Timeouts

​Retrying

​Errors in a stream

​Timeouts

Retrying

Errors in a stream

Timeouts