Updated September 17, 2026. Replaced unsupported latency and abandonment figures, corrected cache accounting, and clarified the tradeoff between long context and retrieval. The original publication date is preserved.
A larger context window is useful capacity. It lets a model consider more material in one request. Whether using that capacity is a good decision depends on what the task needs, how the provider accounts for reuse, and what happens to quality and latency.
The expensive mistake is choosing an input strategy from the advertised window size alone.
Start with an explicit workload
Suppose a support request includes 100,000 input tokens and produces 500 output tokens. At illustrative rates of $10 per million input tokens and $30 per million output tokens, an entirely uncached request costs $1.015.
At 10,000 requests per day for 30 days, that is $304,500. The arithmetic is not a current provider quote or evidence that every support system costs this much. It describes a workload with a large repeated input and no discounts. Change the volume, model or token mix and the result changes.
The useful calculation starts with observed token categories:
monthly cost = sum across requests of (
uncached input × input rate
+ cache writes × write rate
+ cache reads × read rate
+ output × output rate
) / 1,000,000Rates here are per million tokens. Add tool, storage and other platform charges separately. Normalize the provider’s usage schema so cached tokens are not also counted as uncached input. Multiple cache TTLs may have different write rates.
Request hit rate is not cached-token share
An 80% cache hit rate means eight of ten requests found reusable content. It does not mean 80% of all input tokens were read from cache. Prefix lengths and request sizes differ; the remaining requests may pay to write those prefixes.
Your Cache Hit Rate Is Not Your Token Savings works through a batch where 80% of requests hit, 64% of input tokens are cache reads, and input-cost savings are 53.6% at explicitly hypothetical rates.
The Anthropic prompt-caching documentation explains the tools → system → messages prefix, breakpoints, TTLs and usage counters. A timestamp or tool-schema change before the reusable boundary can alter the accounting. Cold parallel requests also need measurement; shared text does not imply that every worker begins with a warm cache.
Measure latency by phase
Longer input can increase prefill work, while output length affects decoding. Dense attention’s quadratic dependence on sequence length describes a part of the computation; it is not a universal formula for whole-request latency. Kernel implementation, model architecture, batching, cache reuse, network conditions and provider scheduling all matter.
Measure time to first token and time to completion separately, with cold and warm cache cases. Include the retrieval and reranking stages when comparing a RAG pipeline with a direct long-context request. An extra retrieval hop may be worthwhile, or may dominate a small task’s latency. There is no single context-length threshold that decides that for every workload.
Avoid inventing user-abandonment percentages from model latency. Product tolerance comes from the workflow and observed user behavior: an interactive autocomplete and an asynchronous research report have different budgets.
Retrieval is another source of error
RAG can reduce input size and keep knowledge updates outside the model. It can also omit the decisive passage, retrieve stale content, or return too little after authorization filtering. Retrieved chunks are not automatically relevant.
Long context can be a good choice when a bounded document set is already known, cross-document relationships matter, and measured quality justifies the cost. Retrieval becomes attractive when the corpus is large, changes frequently, or most queries need a small subset. Many systems combine retrieval with a context window large enough for the selected evidence.
Test answer quality against the same tasks. Record missing evidence, unsupported answers, permission violations and abstentions. A needle-retrieval score alone does not measure whether the application resolves contradictory evidence or performs the requested action.
For multi-tenant retrieval, use an exact baseline inside the authorized corpus. The filtered-search companion explains why a global candidate budget can starve that corpus before the model sees any text.
Make the decision with three measurements
Run a representative sample through a direct-context version, a retrieval version, and any hybrid worth considering. Keep model and task definitions stable where possible. For each, report cost per successful task, latency distribution, and failure categories.
Then stress the assumptions: cold caches, revised documents, large tenants, long conversations and requests requiring evidence from several places. Include retrieval infrastructure and operational work in the decision, rather than comparing token prices alone.
A larger window creates options. The engineering work is choosing how much context to send, proving that the right evidence arrives, and measuring the cost of a completed task.