Every architecture review that involves documents now contains the same argument. One engineer says retrieval is unnecessary because the model accepts a million tokens. Another says stuffing context is a luxury that will not survive the first invoice. Both are arguing about the wrong axis.
Retrieval and long context are not competing products. They are two ways to move the same information into the model's prefill, with different cost curves, different failure modes, and different security properties. The design question is which one your workload's token shape, freshness requirements, and permission model actually favor.
Start with the shape of the tokens
Long context wins when the task needs the whole of something. Summarizing a contract, reviewing a diff, or answering questions about a 400-page filing requires the document as a unit; retrieval can only hand over pieces. Retrieval wins when the task needs the right few paragraphs out of a corpus far larger than any window. A support assistant answering questions across 200,000 articles cannot put the corpus in the prompt, so retrieval is not optional there.
The useful distinction is corpus size against task scope. When the relevant material is a bounded object smaller than the window, long context is simpler and usually more accurate. When the corpus is effectively unbounded, retrieval is the only thing that scales. Mixed workloads, which are the majority, need both.
Cost: prefill scales linearly, caching bends the line
Prefill compute grows with the number of input tokens, roughly two floating-point operations per parameter per token. A 70B model processing 250,000 tokens performs on the order of 3.5 times ten to the sixteenth operations before generating a single output token. At an assumed effective throughput of one quadrillion operations per second, that is about 35 seconds of compute for one request, which is why long context is expensive before anyone marks up a token price.
The arithmetic on the invoice follows. Assume an illustrative input rate of $1 per million tokens. A retrieval design that sends 7,000 tokens costs about $0.007 per request; a long-context design sending 250,000 tokens costs about $0.25, or 35 times more. At a million requests a month that is $7,000 against $250,000. Prompt caching changes the second number dramatically, because a reused prefix bills at a fraction of the base rate: at the 0.1 times rate that Anthropic documents for cache reads, the repeated long prompt falls to about $0.025 per request. That is the difference between a hopeless budget and a plausible one, and it depends entirely on whether the prefix is stable enough to cache.
Long context is cheap to build and expensive to run at volume. Retrieval is expensive to build and cheap to run. The crossover is a volume question: below a few tens of thousands of requests a month, the engineering cost of a retrieval pipeline often exceeds the token bill it would save.
Latency: parallel prefill against a round trip
Prefill is highly parallel, which makes long context feel better than its token count suggests. A 250,000-token prompt on an eight-GPU node processes at a few thousand tokens per second, so time to first token lands in the tens of seconds on the first call and near zero when the prefix is cached. Decode is the serial part and gets slower as the key-value cache grows, because each generated token attends over a larger state.
Retrieval adds a different kind of latency: embedding the query, searching an index, optionally reranking, then a much shorter prefill. The total tends to be more predictable and easier to budget. The important operational difference is variance. A retrieval pipeline has a ceiling on prompt size and therefore a ceiling on worst-case latency. A long-context endpoint's worst case grows with whatever document a user uploads, and a 900-page PDF uploaded at 4:55 p.m. becomes an incident.
Accuracy: retrieval filters, long context hopes
The research here is consistent and unflattering to long context. The Lost in the Middle work showed that model accuracy depends on where relevant information sits within a long input: performance is highest when the answer is at the beginning or the end and degrades significantly when it is buried in the middle. RULER evaluated 17 long-context models on tasks beyond simple needle retrieval and found that although nearly all scored almost perfectly on the basic needle-in-a-haystack test, almost all showed large performance drops as context grew, and only half maintained satisfactory performance at 32,000 tokens.
NoLiMa pushed harder by removing the literal word overlap between question and answer that makes needle tests easy, and found that at 32,000 tokens, 11 of 13 models fell below half of their short-context performance. Retrieval sidesteps this by making the prompt small enough that position effects matter less, and by giving you a lever that long context does not have: reranking. Anthropic reported that contextual embeddings plus contextual BM25 cut top-20 retrieval failure rates by 49%, and 67% when combined with reranking, in its own evaluations.
The practical reading is that retrieval's accuracy is measurable in a way long context's is not. Recall at k, reranker lift, and answer accuracy can be tracked separately. A long-context system fails as one opaque step, and the only diagnostic is a bigger evaluation set.
Freshness and permissions
Freshness favors whichever side you can update faster. Updating an index row takes seconds; a long-context system is always fresh by construction because it reads the source at request time. Where retrieval struggles is the window between a document changing and its chunks being re-embedded, re-indexed, and re-validated, which is why change-data-capture pipelines into the index belong in the design from day one.
Permissions favor retrieval. Enforcing access control at query time, by filtering candidates to the documents the requesting user may see, is a natural part of the retrieval step. Long context has to enforce the same boundaries earlier, when assembling the bundle, and it is easy to assemble a bundle server-side with the wrong scope. Neither approach removes the need for per-request authorization; they only differ in where the check lives, and retrieval puts it somewhere you cannot forget.
Security: both designs eat untrusted text
Retrieved documents are untrusted input, and any document can contain instructions aimed at the model. Retrieval does not fix prompt injection; it changes the blast radius, because only retrieved chunks reach the prompt. The compensation is that retrieval gives you a citation trail: every claim can be traced to a chunk, and anomalous retrieved content is visible in logs. Long context has the same injection surface with a larger payload.
Two controls matter in both designs. Strip or escape content that looks like instructions before it enters the prompt, and keep the model's authority low by requiring tool calls to be allow-listed. A model that can only call a search function and a ticket-update function is far less dangerous than one holding a database credential, whichever retrieval architecture sits in front of it.
Comparison table
| Dimension | Retrieval (RAG) | Long context |
|---|---|---|
| Cost per request | Low and bounded by top-k | Linear in tokens, high unless cached |
| Build effort | Chunking, embeddings, index, reranker, evals | Prompt assembly only |
| Freshness | Depends on index pipeline | Always current |
| Accuracy on lookups | Good if recall is good; measurable | Degrades with length and position |
| Accuracy on whole documents | Weak; chunks lose structure | Strong |
| Permissions | Filter at retrieval | Filter at assembly, easy to get wrong |
| Worst-case latency | Bounded by top-k | Grows with user input |
| Best fit | Large corpora, support, code search | Summarization, review, single-document analysis |
The hybrid that usually wins
- Retrieve a generous candidate set, rerank it, and place the highest-scoring chunks at the start and end of the prompt rather than the middle.
- Cache the stable prefix, meaning instructions and repeated reference material, so repeated long prompts bill at the cached rate.
- Route by task: whole-document tasks get long context, corpus-wide questions get retrieval, and the router is a rule, not a model.
- Enforce permissions at retrieval and verify the assembled bundle against the same policy before sending it.
- Measure recall at k and answer accuracy separately, so a quality regression points at chunking or at generation rather than both.
What to do next
- Write down the corpus size and the task scope for each feature. If the corpus is under the window and the task needs the whole document, skip retrieval.
- Estimate monthly requests times prompt length before choosing. The cost difference is large enough to change the design, not just the budget.
- Put the stable instructions first and make them byte-identical across requests, then measure cache hit rate.
- Build a 100-question evaluation set with known correct sources before adding either pipeline.
- Keep the answer interface identical across both paths so you can shift traffic between them as prices and context windows change.