Skip to main content
When a request starts with the same tokens as a recent one, the GPU reuses the work it already did for that beginning instead of reading it again. This happens automatically; there is nothing to turn on. Cached input is cheaper ($0.10 instead of $0.50 per million tokens for Gemma 4 31B) and makes the first token arrive sooner, which matters most for long system prompts, documents and conversations.

​
How to get cache hits

  • Put what stays the same first. System prompt, tool definitions and reference documents go at the start; the part that changes goes at the end.
  • Keep earlier turns unchanged. In a conversation, send the previous messages exactly as before and add the new ones at the end. Editing an early message makes everything after it uncached.
  • Use the same API key for a conversation. fastrun sends the requests of one conversation (the same key, model, system prompt and first user message) to the same GPU server, where its earlier turns are cached.

​
Seeing cache hits

usage.prompt_tokens_details.cached_tokens tells you how many input tokens came from the cache:
The cache is best effort. Entries are dropped when space is needed, and a request can land on a server that has not seen the beginning yet, for example when the usual one is full. Cached tokens are counted in whole blocks of tokens, so the last few tokens of a shared beginning may not count as cached.