What Enterprise Leaders Need to Know About RAG

RAG is the right default, and it is not a guarantee

Retrieval-augmented generation (RAG) is now the standard way to let a large language model answer questions about an organization's own policies, contracts, products and records. The term comes from a paper by Patrick Lewis and colleagues presented at NeurIPS 2020. They paired a language model, whose knowledge is fixed in its weights, with a searchable index of documents it could consult at answer time. The model stops relying only on what it memorized and starts reading what it is given.

For enterprises the attraction is practical. Content can be updated without retraining anything. Answers can point to the passage they came from. And because the system searches on behalf of a specific user, it can, in principle, show each person only what they are entitled to see. Our comparison of RAG, fine-tuning and custom models explains why retrieval, not training, is where enterprise facts belong.

The mistake many leadership teams make is to treat RAG as the fix for hallucination. It is not. RAG relocates the risk. A RAG system can only be as right as the passages it retrieves, as current as the content behind them and as safe as the permissions attached to them. Those are content, search and access problems. Vendors cannot solve them for you, because they depend on how your organization manages its own knowledge.

How RAG answers a question

Every RAG system, whether bought inside a productivity suite or built in-house, follows the same four steps. Each one is a place where an answer can go wrong before the model writes a single word.

The model in the last step gets most of the attention in vendor demonstrations. In practice, most of the quality, security and cost decisions sit in the first three.

What the legal sector learned the hard way

Law is the clearest public test of RAG, because wrong answers end up in court records and researchers can check them against a ground truth.

Without retrieval. In Mata v. Avianca (2023), lawyers in New York filed a brief citing judicial decisions that did not exist. They had been produced by ChatGPT, used as a research tool with no access to a case-law database. When one lawyer asked the chatbot whether the cases were real, it said they were. On 22 June 2023 Judge P. Kevin Castel of the Southern District of New York sanctioned the two lawyers and their firm, jointly, $5,000. A general model answering from memory will produce fluent, specific and false citations.

With retrieval. Legal research vendors responded by grounding their assistants in real case-law databases, and some marketed the results as free of hallucinations. In 2024 researchers at Stanford's RegLab and Institute for Human-Centered AI tested that claim with a preregistered set of 202 legal questions. Their preprint found that the RAG-based tools from LexisNexis and Thomson Reuters hallucinated on between 17% and 33% of queries. Retrieval clearly helped, since the tools hallucinated less than GPT-4 alone, but it did not come close to eliminating errors.

The detail that matters for leaders is why the tools failed. The researchers counted as a hallucination both a false statement and a real source cited for something it does not say. Their analysis traced many errors to retrieval that missed the most relevant authority, to sources that were real but inapplicable (wrong jurisdiction, overturned, from the wrong court), and to reasoning errors over the retrieved text. A citation, in other words, is evidence that something was retrieved, not evidence that the answer is right.

The same pattern appears outside law. In Moffatt v. Air Canada (2024), the airline's website chatbot told a customer he could claim a bereavement discount after travelling. The airline's own bereavement page said the opposite. The British Columbia Civil Resolution Tribunal held Air Canada responsible for what its chatbot said and rejected the idea that customers should have to cross-check one part of the website against another. When an AI assistant and the authoritative page disagree, the organization owns both answers.

The five places RAG fails

A 2024 study by Scott Barnett and colleagues at Deakin University, drawn from three RAG builds in research, education and biomedicine, catalogued seven failure points. For an executive audience they collapse into five, and only one of them is about the model.

1. The answer is not in the content. If the policy was never written down, or lives in someone's inbox, retrieval cannot find it. A well-built system says "I don't know". A poorly built one finds the nearest passage and answers anyway.

2. Search finds the wrong passage. Documents are split into chunks before indexing. Split badly, and a table is separated from its heading or an exception from its rule. Pure semantic search also struggles with product codes, clause numbers and names, which is why most mature systems combine keyword and semantic search. Anthropic, a model vendor, reported in 2024 that combining context-enriched chunks, keyword search and a reranking step cut the rate of failed retrievals in its own experiments by 67%. Treat that as vendor material, but the direction is consistent with independent work.

3. The content is stale or contradictory. RAG faithfully retrieves whatever is in the index, including the superseded 2022 policy sitting next to the current one. Air Canada's problem was, at root, two sources disagreeing. Without ownership, effective dates and retirement of old versions, the system cannot know which document wins.

4. The model misreads what it was given. Even with the right passages in hand, models make mistakes. Research by Nelson Liu and colleagues at Stanford found that language models use information at the start and end of a long input better than information in the middle. Stuffing more passages into the prompt is not a substitute for retrieving the right few.

5. Permissions are wrong, or the content is hostile. This is the failure that turns a quality problem into a security incident, and it deserves its own section.

Permissions and poisoned content: where RAG becomes a security question

A RAG assistant searches on a user's behalf. Microsoft's documentation for Microsoft 365 Copilot is explicit that it only surfaces organizational data the individual user already has at least view permission for. That is the correct design, and it has an uncomfortable consequence: the assistant inherits every permission mistake you have ever made. A salary spreadsheet shared with "everyone" in 2019 was safe in practice because nobody went looking. An assistant that searches everything a user can see will find it. Fixing overshared content is therefore a precondition for enterprise RAG, not a follow-up task.

The second risk is content that carries instructions. Because a RAG system places retrieved text directly into the model's input, anyone who can get text into the index can try to steer the model. In June 2025 Microsoft disclosed and fixed CVE-2025-32711, known as EchoLeak, which Microsoft rated 9.3 out of 10 (critical). Researchers at Aim Security showed that a single crafted email, later retrieved by Microsoft 365 Copilot as context for a user's question, could cause it to send internal data to an outside server with no click from the user. The lesson for leaders is architectural: every document a RAG system can read is a potential input from an attacker, so external email, shared drives and web content need different trust levels from curated policy libraries. Our guide to building trustworthy enterprise RAG systems sets out the controls in detail.

For organizations in Saudi Arabia, the national data classification scheme set by the National Data Management Office (Top Secret, Secret, Confidential and Public) gives a ready-made basis for which content an assistant may index at all, and for whom. Our guide to the NDMO standards covers those obligations. Classification labels that never reach the search index protect nothing.

What RAG really costs

The visible cost of RAG is small. Converting text into the numerical form a search index uses is cheap: OpenAI lists its small embedding model at $0.02 per million tokens as of 21 July 2026. Illustratively, indexing 50,000 documents averaging 4,000 tokens each, about 200 million tokens, would cost around $4.

The real costs sit elsewhere:

• Content remediation. Finding owners, retiring superseded versions, fixing oversharing and converting scanned or badly structured documents. This is usually the largest effort and the one most often left out of business cases.
• Permission synchronization. Keeping the index's view of who can see what in step with the source systems, continuously.
• Generation. Every answer sends the question plus several retrieved passages to the model, so input tokens per query are often many times the length of the question. Cost scales with usage, not with the size of the library.
• Evaluation and monitoring. A maintained set of test questions with known answers, re-run whenever content, models or settings change.

There is also a case for not building retrieval at all. Anthropic's guidance is that a knowledge base under roughly 200,000 tokens, about 500 pages, can simply be placed in the model's prompt, especially when prompt caching reduces the cost of resending it. RAG earns its complexity when content is large, changes often or is restricted by user.

Questions to ask every RAG vendor

Most enterprise RAG will arrive inside products: productivity suites, service desks, legal and HR platforms. The questions below separate a demonstration from a system you can defend.

Ask for evidence on your own content, not a benchmark on someone else's. The Stanford study is a reminder that products marketed as hallucination-free may not be.

What to do in the next 90 days

1. Inventory where RAG already runs. Copilots, chat assistants and "ask your documents" features are often switched on inside licensed products. List them, the content each can reach and who owns it.
2. Fix oversharing before expanding access. Run a permissions review on the sites, drives and repositories an assistant can search, starting with HR, finance and legal content.
3. Name an owner for each knowledge source. Every indexed collection needs someone accountable for currency and for retiring old versions.
4. Build a test set from real questions. Collect 150 to 300 questions users actually ask, with agreed correct answers and source passages. This is the only reliable way to compare vendors and track change. (INVISOM guidance.)
5. Separate trusted from untrusted content. Decide which sources, such as inbound email and web pages, are treated as potentially hostile, and confirm how each assistant handles them.
6. Require "I don't know". Set the expectation, in contracts and in acceptance tests, that the system declines when retrieval finds nothing adequate rather than answering anyway.
7. Choose use cases where checking is cheap. Start where a user can verify a cited answer quickly. Our framework for selecting AI use cases covers how to weigh verification cost.

Frequently asked questions

What is retrieval-augmented generation in simple terms?

RAG is a way of answering questions with a language model that first searches a collection of documents, then gives the model the most relevant passages to answer from. The model's knowledge is supplemented at answer time instead of being retrained, so content can be updated, cited and restricted by user.

Does RAG stop AI hallucinations?

No. It reduces them but does not eliminate them. A preregistered Stanford study of RAG-based legal research tools found they still hallucinated on 17% to 33% of test queries. Errors came from retrieving the wrong or inapplicable sources as well as from the model's reasoning.

Is RAG secure for confidential enterprise data?

It can be, if the system enforces each user's permissions at search time and the underlying permissions are correct. RAG assistants surface whatever a user can access, so oversharing becomes exposure. Retrieved content can also carry injected instructions, as the 2025 EchoLeak vulnerability in Microsoft 365 Copilot showed.

When should we not use RAG?

When the content is small and stable enough to fit in the model's prompt, roughly a few hundred pages, long context with caching is often simpler. RAG is also the wrong tool for teaching a model a consistent format or behaviour, which is what fine-tuning is for.

Sources and further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020.
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Magesh et al., Stanford RegLab and HAI, preprint May 2024; published in the Journal of Empirical Legal Studies, 2025.
• AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, Stanford HAI, 23 May 2024.
• Mata v. Avianca, Inc., Opinion and Order on Sanctions, US District Court for the Southern District of New York, 22 June 2023.
• Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 14 February 2024.
• Seven Failure Points When Engineering a Retrieval Augmented Generation System, Barnett et al., CAIN 2024.
• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., TACL, 2023.
• Data, Privacy, and Security for Microsoft 365 Copilot, Microsoft Learn, accessed 21 July 2026.
• EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System, Reddy and Gujral, September 2025.
• CVE-2025-32711, NIST National Vulnerability Database, published 11 June 2025.
• Introducing Contextual Retrieval, Anthropic, 19 September 2024 (vendor material).
• text-embedding-3-small, OpenAI model documentation, accessed 21 July 2026.