What Is Prompt Caching? Cached Input Tokens

What is prompt caching? It’s an API optimization for applications that send the same long prompt prefix over and over. Instead of processing that repeated context from scratch each time, the provider stores its processed representation and reuses it for later requests.

A common case is a 3,000-token system prompt sent with every request. The first request processes all 3,000 tokens normally, then later requests can reuse the matching prefix while it remains available in the cache window. PE Collective’s prompt caching glossary uses this 3,000-token system-prompt example.

That changes the economics of an application with heavy instructions, few-shot examples, policy text, or large document context. Your model still has to generate a response. It just avoids repeating work it has already done.

TLDR

Prompt caching stores the processed representation of a repeated prompt prefix. Later requests with that same prefix can skip reprocessing it. It works best when stable system instructions, examples, or document context appear before the part of the prompt that changes.

What Is Prompt Caching

Prompt caching is an optimization where an API provider stores the processed representation of frequently repeated prompt prefixes, avoiding redundant computation on later requests.

The processed representation is usually the key-value cache created during the transformer’s forward pass. That phrase sounds more academic than it needs to be. The model reads your prompt token by token and builds internal state that helps it predict what comes next. A cache preserves that state for an eligible repeated prefix.

The important word is prefix.

A cache hit usually depends on sending the same content at the beginning of the prompt, in the same order. If your stable instructions sit at the front and the user’s changing request comes afterward, the provider can reuse the stable portion. If you shuffle the prompt, insert changing data near the top, or alter the cached text, you may lose the hit.

Cached input tokens are the tokens in that reused prompt prefix. They are input tokens, but the provider does not need to run the same full processing path for them again. They are not generated output. They are not a shortcut around model reasoning. They are prior prompt work the provider can reuse.

That distinction matters when you evaluate latency. A cached prompt can reduce the delay before the model starts producing an answer, but it does not make every part of the request free or instant. New input still needs processing. Output still needs generation. Tool calls, retrieval, and application logic still take however long they take.

The value comes from removing repeated prefix work from the hot path.

How Cached Input Tokens Work

Think about a production assistant that receives a different customer question every time, but always starts with the same instructions. It may include product rules, tone guidance, compliance constraints, examples, and a chunk of reference material.

On the first request, the provider processes the full prefix. That request creates the cacheable internal representation. The application then sends a later request with the same prefix and a new user message appended at the end.

The provider compares the eligible beginning of the later prompt with what it has cached. When the prefix matches and the cache entry is still valid, it can reuse the stored representation instead of processing those tokens again.

The request still contains two kinds of input:

Input type	What happens	Why it matters
Cached prefix	Reused from the prior processed representation	Reduces repeated prompt work
New suffix	Processed for the current request	Carries the user’s new request
Generated output	Produced after the input is understood	Determines the final answer

This structure creates a design constraint that catches teams later than it should. Put stable material first. Put volatile material last.

A system prompt is a strong candidate because it often changes rarely. Few-shot examples can work too, provided you do not rotate, personalize, or randomly reorder them on every request. Large document context can be valuable when many users ask different questions about the same material.

The same logic applies to retrieval-augmented generation, though retrieval can make cache behavior messy. A [RAG glossary]( /glossary/rag/ ) may help frame the architecture, but the practical issue is simpler: an 80% reduction in time-to-first-token only applies when the repeated portion stays repeated. The prompt caching glossary states that subsequent cached requests can reduce time-to-first-token by 80%.

Do not treat caching as a feature you switch on and forget. It is a prompt-layout decision. The application has to preserve a stable prefix long enough for reuse to occur.

A Repeated 3,000-Token Prompt Example

Say your application sends a 3,000-token system prompt with every customer request. That prompt includes operating rules, company terminology, escalation policies, formatting instructions, and examples of acceptable answers.

The first request processes all 3,000 tokens normally. The provider builds the internal representation for that prompt as part of understanding the request. PE Collective’s example says the first request processes all 3,000 tokens normally.

Now a customer asks a question. The assistant receives the full system prompt, the customer’s message, and any other context your application adds. There is no reuse yet because the provider has not processed this exact prefix inside the active cache window.

A later customer asks a different question. Your application sends the same 3,000-token system prompt first, then appends the new customer message. If the repeated prefix matches and remains eligible, the provider reuses the cached representation of the system prompt.

The second request is not identical to the first. It should not be. The user message changes. The model’s response changes. The reusable portion is the stable material at the front.

That is why prompt caching is useful for support assistants, coding copilots, document analysis tools, and internal knowledge applications. These products often carry a large amount of fixed context into each interaction. Without a cache, every request pays to reread it.

With a cache, the model can start from the processed prefix and focus on the current suffix.

This is also where teams accidentally sabotage their own design. A timestamp placed inside the system prompt changes the prefix. Per-user instructions inserted before the stable policy text change the prefix. A dynamically selected example set changes the prefix. Even harmless formatting changes can prevent reuse if the provider needs an exact match.

Treat the prefix like an interface. Keep it stable. Version it deliberately. Make edits in one place. If a change is necessary, expect a cold request while the new version becomes cacheable.

The cost impact can be material. The page states input token costs can fall by 50-90% with prompt caching. A Chain-of-thought glossary is relevant here because the 50-90% range applies to repeated input processing, not to every token involved in an AI response.

The upper end requires a workload with a large reusable prefix and relatively little new input. If each request includes a massive new document, cache savings will be smaller because most of the prompt is still fresh work. The cache does not rescue an application that changes nearly everything on every call.

Cache Windows Change the Implementation

A cache window is the period during which the provider can reuse a processed prefix. It is the practical boundary between a warm request and a cold one.

Your application should not assume that a prefix remains cached indefinitely. Cache availability can depend on provider behavior, traffic patterns, deployment choices, model configuration, and the shape of the request. Build your system so a miss is normal, not exceptional.

A cache miss does not mean the application failed. It means the provider processes the prompt normally, then may make that prefix available for a later reuse. The user still gets an answer. You just do not get the latency or input-cost benefit on that request.

This is why request timing matters. A feature that sends repeated prompts in a tight burst may see frequent hits. A feature that receives sporadic requests may see fewer. The decision depends on whether prompt caching exists in your stack. The right question is whether your traffic pattern creates enough repeated work inside the cache window.

You can improve the odds by grouping stable context into a clear prefix and avoiding unnecessary prompt churn. That does not mean freezing everything forever. It means separating what changes rarely from what changes per request.

A useful prompt structure might look like this:

• Stable product and policy instructions
• Stable examples and formatting rules
• Stable reference context shared across requests
• Current user message
• Current session details or retrieved material that changes frequently

The earlier sections are candidates for reuse. The later sections are the live payload.

This structure also makes debugging easier. When costs rise or latency worsens, you can inspect whether the stable prefix changed. Prompt assembly that looks elegant in application code can be expensive in production if it rebuilds supposedly fixed context in slightly different ways.

When Prompt Caching Helps Most

Prompt caching helps when your application repeatedly sends a large, stable prefix before a smaller changing request.

Large system prompts are the obvious case. Teams often build them up over time: policy rules, product vocabulary, edge cases, safety instructions, response formatting, and examples. None of that is wrong. Sending it anew with every request is where the bill starts to get ugly.

Few-shot prompting is another fit. If your application relies on a fixed set of examples to steer model behavior, those examples belong in the reusable prefix. Keep them consistent. Avoid selecting them randomly unless the quality gain outweighs the lost cache reuse.

Document-heavy workflows can benefit too. Imagine an analyst repeatedly querying the same contract, research report, repository, or policy manual. Put the stable document context before the analyst’s changing question. The model still processes each new question, but it does not need to reread the same shared material from scratch.

An Anthropic API review is worth considering alongside this 90% upper-end input-cost reduction because provider support, pricing treatment, and cache rules determine whether a prompt design produces a real saving. The cost example gives a 90% upper-end input-cost reduction.

Prompt caching is a poor fit when every request is mostly unique. A consumer chat application with short system instructions and long, unrelated user messages may have little reusable context. A workflow that injects freshly retrieved documents ahead of every prompt may also see limited value unless the retrieved material itself repeats.

Personalization can create another tradeoff. If every user gets a different policy block, account profile, or set of examples at the beginning of the prompt, the application creates a separate cache path for each variant. That can still work for high-volume accounts or long-lived sessions. It just will not behave like a single shared prefix.

The workload decides the value.

Prompt Caching Is an Architecture Choice

Prompt caching looks like a pricing feature until you implement it. Then it becomes an architecture choice.

The model provider needs a reusable prefix. Your application needs to create one consistently. Product teams need to decide which instructions are global, which are account-specific, and which belong only to the current request. Engineers need enough observability to distinguish warm requests from cold ones.

That work pays off beyond caching. A clean separation between stable policy, reusable context, and live user input makes prompts easier to audit and update. It also reduces the temptation to keep stuffing every new requirement into a single mutable block of text.

There is a limit, though. Do not contort a prompt solely to chase cache hits. If changing context improves answer quality, change it. A cheap wrong answer is still wrong. Caching should support a sound prompt design, not dictate it.

The better approach is to find repeated work that already exists. If your product sends the same long instructions, examples, or document context across many requests, put that material in a stable prefix and let the provider reuse it when possible.

The model then spends more of its input work on what is new: the customer’s question, the current task, and the context that actually changed.

Key Takeaways

• Prompt caching stores the processed representation of repeated prompt prefixes for later reuse.
• Cached input tokens are reused input context, not generated output tokens.
• Stable instructions, few-shot examples, and shared document context belong at the front of the prompt.
• Cache windows mean a warm request depends on both an exact repeated prefix and request timing.
• The biggest gains come from workloads with large repeated context and smaller changing suffixes.

Sources

• PE Collective prompt caching glossary, 3,000-token system prompt
• PE Collective prompt caching glossary, 80% time-to-first-token reduction
• PE Collective prompt caching glossary, 50-90% input token cost reduction