Key takeaways
- Long context windows let you paste a lot of material into one request. They do not remove the need for retrieval in most business systems.
- Long context is great for one-off analysis of a few documents. RAG is better for large, changing collections, citations and access control.
- Every extra token costs money and time, and models can miss details buried in very long inputs, so sending everything is rarely efficient.
- A hybrid often works best: retrieve the most relevant material, then give the model enough context to reason over it.
- Decide using your data size, how often it changes, who may see what and what each question costs.
The question
Language models can now accept much longer inputs than a couple of years ago, with some offering windows of hundreds of thousands of tokens or more. That prompts a fair question: if the model can read the whole handbook, why build a retrieval pipeline? The answer depends on what you are building.
What each approach does
- Long context. You include the documents directly in the request, and the model reads them all each time.
- RAG. You search your documents for the passages that matter for this question and include only those. See our explainer on what RAG is.
Side-by-side
| Long context | RAG | |
|---|---|---|
| Set-up effort | Low: paste or attach the files | Higher: ingestion, search and evaluation |
| Best for | Analysing a few documents in one go | Large or changing knowledge, repeated questions |
| Cost per question | Grows with the amount of text sent each time | Lower, since only relevant passages are sent |
| Speed | Slower with very long inputs | Faster for most questions |
| Citations | Possible, but you must ask for them | Natural: each answer links to its source passages |
| Permissions | Hard: anything in the prompt is visible to the model | Filter by who is asking at retrieval time |
| Freshness | You resend the latest version yourself | Re-index changed documents |
| Accuracy | Details in the middle of very long inputs can be missed | Depends on the quality of retrieval |
When long context is the right choice
- You need to analyse, compare or summarise a handful of documents, such as a contract and its amendments.
- The task is occasional, so the cost of a large request does not add up.
- You are prototyping and want an answer without building infrastructure.
When RAG is still the right choice
- Your knowledge is large: thousands of documents, tickets and pages.
- It changes often, and answers must reflect the latest version.
- People ask many questions per day, so per-question cost and speed matter. See reducing LLM API costs.
- Different users may see different documents, so you need permission-aware retrieval.
- You want clickable citations and a way to measure retrieval quality.
A hybrid in practice
The two techniques complement each other. A common design retrieves the 10 to 30 most relevant passages and gives the model generous context around them, so it can reason across sources instead of seeing a thin slice. Longer windows also let you include more conversation history and tool results, which is part of good context engineering.
The best way to decide is to test both on a set of real questions and compare accuracy, speed and cost. If you would like help running that comparison on your own documents, our RAG development team can do it for you.