Verdict first: the KV cache speeds up the single request you are already running, prefix caching reuses that work across requests sharing the same opening tokens, prompt caching turns that reuse into a vendor-reported billing discount, and semantic caching skips the model entirely on paraphrased repeats at the cost of threshold tuning you must own. The four layers stack rather than substitute: keep the KV cache healthy, structure prompts so prefix reuse can fire, let the provider discount the stable prefix, and add a semantic layer only where paraphrase repeats justify a vector store. This guide expands the memory story from the KV cache entry and the framing story from the system prompt entry, with sizing context from the quantization guide and embeddings on consumer GPUs, first covered alongside the September 21 digest.
Layers at a glance
The grid below summarizes the four layers. The first three reuse exact token prefixes and are correctness-neutral; the fourth matches by meaning and can return a wrong answer with a normal success status (Daily Dose of Data Science, August 28, 2026; Traversaal, September 7, 2026).
| Layer | Stack position | Resource saved | Scope | Needs custom infra |
|---|---|---|---|---|
| KV cache | Inference engine, GPU memory | Attention recomputation within one request | Single request | No, built into every serving engine (packet.ai KV cache guide) |
| Prefix caching | Inference server block manager | Prefill compute on shared opening tokens | Cross-request, exact prefix match | No, server configuration only (packet.ai prefix caching guide) |
| Prompt caching | Provider billing boundary | Billed input tokens on the cached prefix | Cross-request, exact prefix match | No, provider feature with breakpoints (explainx.ai prompt caching guide) |
| Semantic cache | Application layer, vector store | Full inference call, input and output tokens | Cross-request, similarity match | Yes, embeddings plus vector index and TTL policy (Traversaal caching strategies guide) |
KV cache: the per-request speed lever
Each attention layer computes key and value tensors per token; the KV cache keeps them so every new token attends to stored history instead of recomputing the whole sequence (packet.ai KV cache guide). It never lowers a hosted bill by itself: it buys faster decoding inside one call at the price of GPU memory that grows with context length times concurrent sessions. On your own hardware the same tradeoff appears as context-size settings: larger contexts hold longer sessions but reserve more memory per slot, so parallel request capacity drops, exactly the sizing math in the quantization guide. Local-runner note: no flag is needed to benefit, but size the run so the cache fits — short --max-model-len windows and quantized weights leave more room for concurrent sessions than a full context you never fill.
Prefix caching: cross-request reuse on your server
Prefix caching keeps finished KV blocks indexed so a later request with a byte-identical opening prefix reuses them instead of recomputing prefill (packet.ai prefix caching guide; Daily Dose of Data Science, August 28, 2026). Blocks are matched by a chained hash over token IDs, so any reordering, timestamp, or user ID injected ahead of the stable content breaks the whole match silently. Local-runner notes, verified against current upstream sources during implementation: in vLLM, automatic prefix caching is enabled by default on the current engine (enable_prefix_caching: bool = True in the engine cache configuration), controlled from the command line with --enable-prefix-caching and explicitly disabled with --no-enable-prefix-caching, with the block hash selectable via --prefix-caching-hash-algo; in llama.cpp, reusable prompt state is managed with --prompt-cache FNAME plus --prompt-cache-all, and --keep N controls how many initial-prompt tokens are retained on context reset. Stable-content-first layout is the rule in both: system instructions, tool definitions, and frozen reference documents up front in a fixed order, conversation history and per-request values at the end.
Prompt caching: the provider-billed version of prefix reuse
Prompt caching is a hosted provider running prefix reuse on its own hardware and discounting the reused input tokens, with the cached object still being KV tensors behind an exact-prefix match (Daily Dose of Data Science, August 28, 2026; explainx.ai prompt caching guide). Vendor-reported economics, labeled as vendor-reported and subject to change: Anthropic charges a vendor-reported 1.25x premium on the base input rate to write a cache entry and a vendor-reported 0.1x rate to read it, with a higher write multiplier on longer retention, and current OpenAI models apply the same pair of vendor-reported multipliers (Daily Dose of Data Science; explainx.ai system prompt guide). Check the provider price page before budgeting; no figure here is a quote. Provider mechanics differ: Anthropic uses explicit cache_control breakpoints marking the end of the stable prefix, while OpenAI-style caching is automatic above a minimum prefix size (explainx.ai prompt caching guide). The layout rule from the system prompt entry applies directly: never interpolate dates, user IDs, or mode flags into the cached prefix; inject dynamic context in later messages instead.
Semantic cache: skipping the call by meaning
A semantic cache embeds the incoming prompt, runs a nearest-neighbor search over stored prompts, and returns a stored response when similarity clears a threshold — saving input and output tokens together because the model never runs (Traversaal, September 7, 2026). That is also why it is the only layer here that can be confidently wrong: lower the threshold and paraphrase coverage climbs alongside the false-positive rate, while every request pays an embedding round trip including every miss. Threshold and TTL pitfalls: published defaults range widely (roughly 0.75 to 0.97 depending on who you ask), which means the threshold is a property of your traffic, not a value to copy — start near 0.90 to 0.95 on low-stakes FAQ-like traffic, replay real query pairs before widening, exclude personalized, stateful, and time-sensitive categories outright, and give volatile entries short TTLs with event-driven purges rather than one global TTL. Embeddings for the index itself run fine on local hardware; see embeddings on consumer GPUs for the sizing.
Honest limits
Three limits keep this guide honest. First, the 1.25x-write and 0.1x-read multipliers are vendor-reported figures carried from secondary guides to provider price pages, not measurements from this site; verify them against the live provider pricing before making any budget claim. Second, exact-prefix layers fail silently on structure: a single reordered document, a timestamp at the top, or a growing history inserted mid-prefix collapses hit rates toward zero with no error, so prefix discipline is an ongoing deployment artifact, not a one-time setting. Third, semantic caching trades correctness for coverage in a way the other layers never do — a wrong cached answer returns with a normal success status — so high-stakes domains should keep thresholds near exact-match territory or skip the layer, and every layer needs its own hit-rate and quality logging to avoid double-counting savings when two layers fire on the same request.
Questions and answers
What is the difference between KV cache, prefix caching, prompt caching, and semantic cache?
The KV cache stores attention tensors for one request; prefix caching keeps those tensors on the server for later requests with the same opening tokens; prompt caching is the provider-billed version of that reuse with a vendor-reported discount; a semantic cache stores finished responses keyed by embedding similarity and skips the model on paraphrase matches.
When does provider prompt caching make a semantic cache redundant?
Rarely. Prompt caching discounts matched input prefixes at the billing boundary while semantic caching eliminates the whole call at the application layer; they target different cost drivers and compound when stacked, with the semantic layer checked first and the provider prefix firing on whatever falls through.
How should local runners structure prompts for prefix hits?
Put stable content first in a fixed order — system instructions, tool definitions, frozen documents — and dynamic content last. Keep vLLM prefix caching enabled unless every prompt is unique or isolation demands it, reuse llama.cpp prompt-cache files across sessions sharing the same initial prompt, and confirm behavior with the server hit counters rather than assuming.
What similarity threshold should a semantic cache use?
Whatever your false-positive budget allows on your own query pairs. Start conservative near 0.90 to 0.95 on low-stakes traffic, measure hit rate against wrong-answer rate side by side, and exclude legal, medical, financial, and other high-stakes categories until replay testing justifies inclusion.
Where do I go next on this site?
Read what the KV cache is and what a system prompt is, size your card with the quantization guide, plan the embedding side with embeddings on consumer GPUs, then catch the wider context in the September 21 digest.