How to Build Trustworthy Enterprise RAG Systems

Build from the index outward

A retrieval-augmented generation (RAG) system is trustworthy when three things hold for every answer: the user was entitled to every passage the model saw, the passages were the right ones, and the answer says only what those passages support. The first is a security property, the second a search property, the third a generation property. Only the third depends much on the model.

Yet most RAG projects spend their effort the other way round, iterating on prompts and comparing models while the index is built once and left alone. The evidence points the other way. In Stanford's 2024 study of RAG-based legal research tools, many hallucinations traced back to retrieval that missed the most relevant source or returned a real but inapplicable one. In Microsoft's own retrieval experiments, changing the search configuration moved relevance scores by roughly 20 points on a 100-point scale. A model upgrade rarely does that. And a permission leak or an injected instruction cannot be fixed by a better prompt after the fact.

This guide sets out a build order: ingestion and access control first, then retrieval quality, then citations and generation, with evaluation as the release gate and lineage and monitoring around all of it. The executive view of why RAG fails is in our companion piece, What Enterprise Leaders Need to Know About RAG. The choice between RAG and fine-tuning is covered in RAG, fine-tuning or a custom LLM.

A reference architecture

The architecture is vendor-neutral. Each layer can be built with a managed search service, an open-source engine or a database with vector support. What matters is that every layer carries identity, version and provenance forward, so that any answer can be traced back to the documents, permissions and settings that produced it.

Ingestion: the index is a data product

Treat the index as a governed data product with an owner, a schema and a service level, not as a cache. At ingestion, capture five things for every chunk:

• Source and version. A stable document ID, the version or content hash, and the source URI. This is what makes citations and lineage possible.
• Effective dates and status. When the document took effect, when it was superseded and whether it is draft or approved. Superseded content should be excluded or down-ranked by default.
• Owner. The person or team accountable for keeping the source current.
• Classification. The sensitivity label from your scheme. In Saudi Arabia, the National Data Management Office's levels (Top Secret, Secret, Confidential, Public) are the natural basis; our NDMO implementation guide covers how they are assigned.
• Access control. The users or groups allowed to read the source, copied onto every chunk.

Parsing deserves more care than it usually gets. Tables, headings, numbered clauses and footnotes carry meaning that plain-text extraction destroys. Scanned PDFs need OCR with a quality check. Where the structure matters, parse it explicitly rather than hoping the chunker preserves it.

Access control has to happen at retrieval

The rule is simple: filter by the user's permissions inside the search query, before ranking, never after generation. If unauthorized passages reach the prompt, the model has seen them, and no output filter can reliably stop them leaking into the answer.

Three implementation points are easy to miss:

• Chunks must inherit permissions. Microsoft's documentation for Azure AI Search notes that when content is split into chunks, sensitivity labels must be projected onto each chunk row; without that, chunk-level results are not filtered. The same applies to any engine.
• Permission sync has latency. Query-time enforcement can only use the permission metadata already in the index. Microsoft states plainly that changes in the source system appear in results only after the next synchronization. Measure that lag, set a target for it and treat revocations of access to sensitive content as urgent.
• A filter string is not authorization. The common "security filter" pattern matches a group ID string against a field. Microsoft describes it as simulating document-level authorization by string comparison. It works, but only if the identity passed in comes from a verified token, never from the client.

RAG also exposes every historical permission mistake, because it searches everything a user can see. Run an oversharing review of the sources before connecting them.

Chunking and retrieval quality

Chunk small, keep structure

Microsoft published unusually detailed retrieval experiments for Azure AI Search in 2023, run on customer datasets and public benchmarks. Two findings generalize. First, chunking long documents substantially improved recall compared with one vector per document. Second, smaller chunks retrieved better than large ones.

These are vendor experiments with one embedding model, so treat the exact numbers as indicative. The practical defaults are chunks of a few hundred tokens with some overlap, boundaries at headings and sentences rather than fixed character counts, and the document title and section path attached to each chunk. Anthropic reported in 2024 that prepending a short generated summary of each chunk's context before embedding cut failed retrievals by 35% in its own tests. Vendor material again, but cheap to test on your own set.

Use hybrid search and a reranker

Vector search matches meaning; keyword search (typically BM25) matches exact terms. Enterprise questions need both, because they mix concepts with policy numbers, product codes, clause references and names.

The keyword-query row is the one to remember. On queries built around exact terms, Microsoft measured vector search at 11.7 against 79.2 for keyword search. A vector-only system will fail exactly the questions where users expect precision.

Merge the two result lists with reciprocal rank fusion, a simple method published by Cormack, Clarke and Büttcher in 2009 that combines rankings without needing comparable scores. Then pass the top few dozen candidates to a cross-encoder reranker, which reads query and passage together and reorders them. In Microsoft's tests the reranking step added more than the move from vector to hybrid. Retrieve broadly, then rerank and keep only the best few passages for the prompt: models use the start and end of long inputs better than the middle, so more context is not better context.

Citations and generation

Citations are the user's main defence, and they must be precise enough to check:

• Cite at chunk level with version. Link to the passage and the document version that was retrieved, not to a document's home page.
• Check support, not just presence. The Stanford study counted a real source cited for a claim it does not make as a hallucination. Add an automated check, either a second model call or a rules-based comparison, that each cited sentence is supported by its passage, and suppress or flag claims that fail.
• Make declining a feature. Instruct the model to answer only from the provided passages and to say so when they are insufficient. Set a retrieval score threshold below which the system declines without calling the model. Track the decline rate; a rate near zero usually means the system is guessing.

Evaluation is the release gate

No change to content, chunking, embeddings, retrieval settings, prompts or model should reach production without passing an evaluation run. That requires a golden set and a scorecard.

The golden set. Start with 150 to 300 real user questions with agreed answers and the passages that support them. Include questions that should be declined, questions that rely on recently changed content, exact-term questions, and questions asked by users with different permissions. For Gulf deployments include Arabic, English and mixed-language questions. (INVISOM guidance; there is no standard size.)

The metrics. Open-source frameworks such as Ragas, introduced in a 2023 paper by Shahul Es and colleagues, provide automated metrics including context precision, context recall, faithfulness and response relevancy. These use a language model as judge, so calibrate them against a sample of human-graded answers before trusting the scores, and re-check when the judge model changes.

The two security rows are pass/fail. A single permission leak in testing is a release blocker regardless of how good the accuracy scores look.

Lineage and monitoring

For each answer, log enough to reproduce it: the question, the user's identity or a pseudonymous reference, the retrieved chunk IDs with document versions, the index version, the prompt template version, the model and its version, and the answer. Apply your retention and personal-data rules to the log itself, since it will contain both. Our article on moving AI from pilot to enterprise value explains why this evidence also matters for proving outcomes.

Upstream, record which source systems and pipeline runs produced each index version. Open standards such as OpenLineage, a graduated project of the LF AI & Data Foundation, define a common model of jobs, runs and datasets that ingestion pipelines can emit, so index builds appear in the same lineage graph as the rest of your data estate.

In production, monitor the decline rate, the zero-result rate, the share of answers citing superseded documents, permission sync lag, user feedback, and evaluation scores on a scheduled re-run of the golden set. A sudden change in any of them usually means the content changed, not the model.

Prompt injection through retrieved content

The OWASP Top 10 for LLM Applications ranks prompt injection first (LLM01:2025) and distinguishes direct injection, typed by the user, from indirect injection, carried in external content the model processes. OWASP notes that RAG does not remove the vulnerability. In a RAG system, every indexed document is potential attacker input.

EchoLeak (CVE-2025-32711), disclosed and fixed by Microsoft in June 2025, is the reference case. A single crafted email, retrieved by Microsoft 365 Copilot as context, led it to exfiltrate internal data without any user action. The researchers' write-up shows a chain of bypasses, and each link maps to a control.

Three principles follow. Tier your sources: curated policy libraries, internal collaboration content and external content such as inbound email or web pages warrant different trust and, often, separate indexes. Mark untrusted text: Microsoft researchers reported that "spotlighting" techniques, which transform or delimit retrieved content so the model can tell data from instructions, reduced attack success from over 50% to under 2% in their tests on GPT-family models. Limit what output can do: no automatic rendering of external images or links, no tool calls triggered by retrieved text without a human or policy check, and least privilege for any actions the assistant can take.

Arabic and multilingual retrieval

Arabic needs specific handling at both the keyword and vector layers.

Keyword search. Use an Arabic analyzer. Apache Lucene's Arabic normalizer, used by engines built on Lucene, unifies hamza forms on alef, converts teh marbuta to heh and alef maksura to yeh, and strips diacritics and the tatweel stretching character. Without this, the same word spelled two common ways will not match. Pair normalization with light stemming, and test it on your own content, because aggressive stemming can merge terms that matter in legal and regulatory text.

Vector search. Use an embedding model trained for multilingual retrieval. BGE-M3 from the Beijing Academy of Artificial Intelligence, for example, supports more than 100 languages, inputs up to 8,192 tokens, and dense, sparse and multi-vector retrieval from one model. Public multilingual benchmarks such as MIRACL include Arabic, but they reflect Modern Standard Arabic in Wikipedia, not your mix of formal documents, Gulf dialect queries and Arabic-English code-switching.

Evaluation. Report every scorecard metric separately for Arabic questions. Include cross-lingual cases, such as an Arabic question answered from an English policy, and check that citations and quoted passages display correctly right to left. If generation quality in Arabic is the bottleneck rather than retrieval, the regional model options are covered in our adaptation guide.

Personal data. Saudi Arabia's Personal Data Protection Law has been fully enforceable since 14 September 2024. When personal data in a source must be corrected or deleted, the index, any caches and the answer logs are in scope too. Build deletion propagation into the pipeline from the start.

What to do in the next 90 days

1. Write down the three trust properties (entitled, relevant, supported) as acceptance criteria for every RAG system you run or buy.
2. Build the golden set of 150 to 300 real questions, including declines, exact-term, permission-varied and Arabic cases, and agree answers with content owners.
3. Add ingestion metadata: version, effective dates, owner, classification and access control on every chunk. Exclude superseded content.
4. Move permission filtering into the query if it happens anywhere later, and measure permission sync lag.
5. Switch on hybrid retrieval and reranking, then compare against your current configuration on the golden set before changing anything else.
6. Tier your sources and red-team the untrusted tier with indirect injection tests based on the OWASP LLM01 guidance.
7. Make the scorecard a release gate in your deployment pipeline, with the security rows as hard blockers.

Frequently asked questions

What is the most effective way to improve RAG accuracy?

Improve retrieval before changing the model. Hybrid search that combines keyword and vector retrieval, followed by a reranker, and smaller structure-aware chunks produced the largest gains in Microsoft's published Azure AI Search experiments. Measure every change on a golden set of real questions.

How do you enforce document permissions in RAG?

Copy each source document's access control onto every chunk at ingestion, and filter by the verified user's groups inside the search query before ranking. Never rely on filtering the model's output. Monitor how long permission changes take to reach the index.

How do you evaluate a RAG system?

Use a golden set of real questions with agreed answers and supporting passages. Measure retrieval (context recall and precision), generation (faithfulness and citation support), end-to-end correctness and declines, plus pass/fail security tests for permission leakage and prompt injection. Frameworks such as Ragas automate many metrics.

How do you protect RAG from prompt injection?

Treat retrieved content as untrusted input. Separate trusted and untrusted sources, mark retrieved text so the model can distinguish data from instructions, block automatic rendering of external links and images, and require approval before any action triggered by retrieved content. OWASP LLM01:2025 lists the core mitigations.

Sources and further reading

• Azure AI Search: Outperforming vector search with hybrid retrieval and reranking, Microsoft, 18 September 2023 (vendor material).
• Document-level access control in Azure AI Search, Microsoft Learn, accessed 26 July 2026.
• Security filter pattern in Azure AI Search, Microsoft Learn, accessed 26 July 2026.
• Introducing Contextual Retrieval, Anthropic, 19 September 2024 (vendor material).
• Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, Cormack, Clarke and Büttcher, SIGIR 2009.
• Ragas: Automated Evaluation of Retrieval Augmented Generation, Es et al., 2023, and the Ragas metrics documentation.
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Magesh et al., Stanford RegLab and HAI, 2024.
• LLM01:2025 Prompt Injection, OWASP Gen AI Security Project.
• EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System, Reddy and Gujral, September 2025.
• Defending Against Indirect Prompt Injection Attacks With Spotlighting, Hines et al., Microsoft, 2024.
• OpenLineage documentation, LF AI & Data Foundation.
• ArabicNormalizer, Apache Lucene API documentation.
• M3-Embedding (BGE-M3), Chen et al., 2024.
• MIRACL, multilingual retrieval benchmark covering 18 languages including Arabic.