Prompt Caching vs. Intelligent Prompt Routing: Two Strategies for Cutting Bedrock AI Inference Costs
Prompt Caching vs. Intelligent Prompt Routing: Two Strategies for Cutting Bedrock AI Inference Costs
Written by Minhyeok Cha
Table of Contents
Why is AI inference cost a problem right now?
Prompt Caching
Intelligent Prompt Routing
What to consider when running them together
Wrapping up
How much attention do you actually pay to token costs when running Bedrock? During development it never feels real — you make a handful of test calls and move on. But once you ship to production, tens of thousands of API calls pile up every day, and every single input token turns into money. To tackle exactly this problem, AWS offers two headline tools: Prompt Caching and Intelligent Prompt Routing.
In this post, we'll walk through how each feature works, when to reach for one over the other in practice, and what to watch for when you run them together. Whether you're designing a Bedrock architecture from scratch or already in production and seeing bills higher than you expected, we hope this gives you a framework for making the call.
Why is AI inference cost a problem right now?
Gartner figures show that enterprise GenAI spending grew 76.4% year over year in 2025 to reach $644 billion, with roughly another 80.8% of growth projected for 2026. That number isn't just "the AI industry is big." A significant share of that spending is inference cost billed per token.
Bedrock calculates and bills input tokens and output tokens separately on every call. The trap is token amplification. A single user question can trigger multiple model invocations internally, and in agent workflows it's common to consume 4–8x more tokens than the actual input and output would suggest. You start out thinking "how much could it really cost?" — and the moment production traffic hits, the bill lands at several times what you estimated.
Why does this happen? In development, you only make a few test calls, so nothing feels urgent, and a long system prompt doesn't bother you. But when that same long prompt is fed in tens of thousands of times a day, or when every request — regardless of complexity — is aimed at your most expensive model, costs accumulate quietly and quickly.
The structured way to approach this is the two features we'll cover in the next sections: Prompt Caching and Intelligent Prompt Routing.
Prompt Caching
Without caching, every API call reprocesses that static system prompt at full price, every single time. Turn Prompt Caching on, and the structure shifts: the first call stores the prompt once, and subsequent calls read it back at a 90% discount. It's a bit like tallying a receipt once for the first customer, then copying that result for everyone after.
The cost structure is worth spelling out. For third-party models like Claude, the initial cache Write costs about 25% more than the standard input token price. But every time you reuse the cache (Read), you pay only about 10% of the standard input token price — effectively a 90% discount. If your pattern is one Write and thousands of Reads, the math clearly works in your favor.
The thing people most often miss in practice is the TTL (Time To Live) — the cache's lifetime. The default cache lifetime is 5 minutes, and every cache hit extends it by up to 5 more minutes. If your call interval exceeds 5 minutes, the cache expires and you incur the Write cost all over again. In other words, for services with uneven traffic, your cache hit rate can be lower than you'd expect.
Intelligent Prompt Routing
If Prompt Caching is about "how do we pay less for repeated content," Intelligent Prompt Routing starts from a different angle: "does this question really need to go to the expensive model?"
Intelligent Prompt Routing predicts the complexity of a request and automatically routes it to the appropriate model within the same model family. You don't have to build a separate routing layer — the system predicts each model's response quality for a given request and picks the model that optimizes for quality and cost at the same time.
Per AWS's official guidance, it can cut costs by up to 30% without compromising on accuracy, and depending on the benchmark, figures as high as 30–50% have been reported. Of course, the actual savings depend on your workload's characteristics.
The point I personally consider important in practice is the transparency of the routing decision. You can confirm which request went to which model via CloudWatch Logs, but the routing rationale itself is close to a black box. That means predicted quality can diverge from actual quality, so when you first adopt it, it's better to design in A/B testing or quality monitoring alongside it.
Both features share the goal of "reducing AI inference cost," but they attack it along completely different axes. Prompt Caching is a strategy for reducing the number of times tokens are processed; Intelligent Prompt Routing is a strategy for improving the efficiency of model selection. The former addresses "how much of the same content are we processing over and over," the latter "how expensive a model are we using for this request."
Category | Prompt Caching | Intelligent Prompt Routing |
|---|---|---|
Core question | Are we processing the same context repeatedly? | Are we using the same model for every request? |
How it saves | Eliminates repeated input token cost | Routes simple requests to a lower-cost model |
Best-fit workloads | Document Q&A, coding assistants, long system prompts repeated often | Chatbots, customer service, and other conversational services with wide complexity variance |
Developer involvement | Requires marking cache points | Automatic routing after router setup |
Max savings | Up to 90% cost, up to 85% latency | Up to 30–50% cost |
Key constraints | 5-minute cache lifetime, limited model support | Optimized for English prompts, pairwise models only |
The question we get most often in practice is "so do I have to pick just one?" — but they aren't mutually exclusive. You can absolutely apply Prompt Caching to a service with a long, repetitive system prompt while also layering Intelligent Prompt Routing on top if that same service has wide variance in request complexity.
The catch is that the conditions under which each feature pays off differ by workload. So before applying either, the right first step is to figure out which type of loss your own service is incurring more of.
What to consider when running them together
When you apply both features at once, there are a few things to check first.
The first is cache write cost. Many people assume "Prompt Caching always saves money" — but that's not the case. For third-party models like Claude, the cache write cost is set about 25% higher than the standard input token rate. At the moment you first create the cache, you're actually spending a little more. When running Claude on Bedrock, the cache hit-rate break-even point sits at roughly 30%; below that, the cache write cost exceeds your savings.
The second is router model version management. Since Bedrock's GA, Intelligent Prompt Routing lets you directly select the two models you want from a model family and configure the routing criteria yourself. That added flexibility also means more to manage. If you don't check the currently available prompt router configurations in the console beforehand, requests can end up routed to an unintended model combination. Versions don't switch automatically, so it's a good idea to review your router settings each time a new model is released.
The third is TTL and traffic. Claude Haiku 4.5, Sonnet 4.5, and Opus 4.5 let you choose a 5-minute or 1-hour TTL. For a service where traffic concentrates in specific windows, a 1-hour TTL can raise hit rates dramatically — but for a service where requests trickle in sporadically, it becomes a structure where you only keep paying the write cost after the cache expires. Intelligent Prompt Routing's routing decisions are also ultimately made per real-time request, so keep in mind that irregular traffic patterns can degrade routing accuracy.
Wrapping up
In this post, we introduced Prompt Caching and Intelligent Prompt Routing. What matters is the order in which you adopt these two features, and which workloads you apply them to. If your system prompt is long and highly repetitive, caching comes first. If request complexity is uneven and the cost gap between models is large, routing is where the value is.
Personally, when first adopting these two features, I recommend monitoring the CacheReadInputTokens and CacheWriteInputTokens metrics in CloudWatch — along with your actual per-model invocation ratios — for at least two weeks before tuning. The setup is quick, but it takes time for enough of a traffic sample to accumulate before the effect becomes visible.
With AWS Bedrock architecture, how cost-efficiently you can operate comes down less to knowing each individual feature and more to how accurately you understand your own workload's patterns.
