Enterprise RAG Architecture: A Production Reference Design

An enterprise RAG architecture that survives production has eight parts: an ingestion pipeline that chunks by document structure, hybrid retrieval (vector plus keyword), a reranker, document-level access control enforced at query time, answers that cite their sources, an evaluation harness for both retrieval and answers, a freshness strategy, and tracing on every request. The model you pick matters less than most teams expect. Most production failures come from retrieval, permissions and stale data, not from generation.

Below is the reference design we use when scoping enterprise RAG development projects, the trade-offs behind each component, and the failure modes to plan for before launch.

Why demo RAG breaks in production

A typical prototype splits PDFs into fixed-size chunks, embeds them, pulls the top five by cosine similarity and pastes them into a prompt. It works on 50 curated documents. Then it meets the real corpus: near-duplicates, tables cut in half, questions that hinge on a product code embeddings treat as noise, a sales rep shown an HR document, a policy that changed last Tuesday. None of these is a model problem, and a bigger LLM fixes none of them.

Production RAG is a search system with a language model at the end. Design it that way.

The reference architecture at a glance

Every request goes through two paths. The offline path (ingestion) turns source systems into searchable, permission-tagged chunks. The online path (query) resolves the user's identity, retrieves candidates, filters and reranks them, generates a grounded answer with citations, and logs everything.

Ingestion and chunking: where quality is decided

If the right text never makes it into the index in a usable shape, nothing downstream can recover it. Spend real time here.

Parse for structure, not just text

Plain text extraction loses headings, table boundaries and reading order. Use layout-aware PDF parsing and keep HTML or Markdown structure. Keep each table as one chunk, or serialize each row with its column headers so "Plan: Enterprise | Seats: 500 | SLA: 99.9%" makes sense on its own.

Chunk by meaning, then by size

Fixed-size chunking is a reasonable baseline and a poor end state. A better default:

• Split on document structure first: sections, headings, list items, table rows.
• Apply a size ceiling second, so very long sections are split into smaller pieces with modest overlap.
• Prepend the document title, section path (for example "Handbook > Leave > Parental Leave") and effective date to every chunk. This often helps more than switching embedding models.
• Store a pointer to the parent section so you can expand a small matched chunk into its surrounding context at generation time ("small-to-big" retrieval).

Capture metadata at ingest, not later

Every chunk should carry source system, document ID, URL, title, owner, timestamps, version, document type and the access control list. Retrofitting ACLs means re-ingesting everything, so capture them on day one.

Deduplicate

The same policy often lives in SharePoint, Confluence and an email attachment. Without deduplication, your top five results can be five copies of one paragraph. Hash normalized text for exact duplicates, use similarity thresholds for near-duplicates, and prefer the most recent or authoritative source.

Hybrid retrieval: vector plus keyword

Dense vector search is good at meaning ("how do I expense a client dinner" matches "meal reimbursement policy"). It is bad at exact tokens: SKUs, error codes, contract numbers, people's names, and internal acronyms that the embedding model has never seen. Keyword search (BM25) has the opposite profile. Enterprise questions contain both kinds, often in the same query.

Run both in parallel and merge the results. Reciprocal rank fusion (RRF) is the common starting point because it only needs ranks, not comparable scores:

def reciprocal_rank_fusion(result_lists, k=60):
    """Merge several ranked lists of document IDs into one ranking."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

fused = reciprocal_rank_fusion([vector_hits, keyword_hits])

Retrieve generously here (say 50 to 100 candidates), because the reranker will narrow them down. OpenSearch, Elasticsearch and several vector databases support hybrid queries natively.

Once the basics work, add query rewriting, which turns a follow-up like "what about for contractors?" into a standalone query. If you are still choosing tooling, our overview of popular RAG frameworks covers the libraries that implement these patterns.

Reranking: the cheapest big win

First-stage retrieval optimizes for recall and speed. A reranker, usually a cross-encoder that reads the query and each candidate together, optimizes for precision. It takes your 50 to 100 fused candidates and returns the best 5 to 10 for the prompt.

Reranking is usually the highest-return addition to a basic pipeline, because the common failure is not "the right chunk wasn't found" but "it was found and ranked 14th." The cost is latency, so cap the candidate count and measure it. Hosted rerank APIs and open-source cross-encoders both work; choose based on data residency and volume.

Permissions and document-level access control

This is the component that decides whether security signs off. The rule is simple: a user must never receive an answer built from a chunk they could not open in the source system.

Enforce at retrieval, not after generation

Filtering the answer after generation does not work: once restricted text is in the prompt, it can leak in paraphrased form. Filter candidates before they reach the model:

• Pre-filtering with ACL metadata. Store allowed users and groups on each chunk. At query time, resolve the user's identity and group memberships from your identity provider and pass them as a filter to both the vector and keyword queries. This is the default for most enterprise deployments.
• Separate indexes per tenant or security domain. Stronger isolation, simpler to reason about, and a good fit for multi-tenant SaaS or strictly separated business units. It gets expensive when permissions are fine-grained.

Post-filtering (retrieve top K, then drop what the user cannot see) is tempting but fragile: if a user can see few documents, most of the top K gets dropped and recall collapses. If you must post-filter, over-fetch heavily.

Keep permissions in sync

ACLs change more often than content. Sync permission changes on their own, faster schedule, and treat a failed ACL sync as a paging alert, not a log line. Cache keys for answers must include the user's permission scope.

Citations users can verify

Citations turn "the AI said so" into "the policy says so, here." They also make evaluation and debugging far easier. Practical guidance:

• Pass chunks to the model with stable IDs and ask for structured output that references those IDs per claim or per paragraph.
• Validate citations in code: every cited ID must exist in the retrieved set. Drop or flag answers that cite nothing.
• Render citations as links to the exact source location (page, section anchor or record URL), not just the document title.
• Instruct the model to say it does not know when the retrieved context does not support an answer, and test that behavior explicitly. A confident wrong answer with a citation attached is worse than no answer.

Evaluation: measure retrieval and answers separately

Without an evaluation harness, every change is a guess. Build it before you tune anything.

Build a golden set

Collect 100 to 300 real questions from the people who will use the system, each paired with the source documents that should answer it and, where practical, a reference answer. Include hard cases: questions requiring two documents, questions with exact identifiers, questions the corpus cannot answer, and questions a given role should be denied.

Retrieval metrics

• Recall@K: did the correct source appear anywhere in the top K candidates?
• MRR or nDCG: how high did it rank?
• Permission correctness: did any restricted chunk appear for a user who should not see it? The target is zero.

Answer metrics

• Faithfulness (groundedness): is every claim supported by the retrieved context?
• Answer relevance and correctness: does it answer the question, and does it match the reference?
• Citation accuracy: do the citations actually support the claims they sit next to?
• Abstention: does it decline when it should?

LLM-as-judge scoring scales well, but calibrate it against human ratings first. Run the suite in CI on every change. When recall is low, fix ingestion and retrieval; when recall is high but answers are wrong, fix the prompt or model.

Freshness: keeping the index honest

Users lose trust fast when the assistant quotes last quarter's price list. Design freshness per source:

• Use change feeds, webhooks or modified-since queries for incremental updates instead of full re-crawls.
• Handle deletes explicitly. A document removed from the source must be removed from both indexes; tombstone handling is the part most pipelines forget.
• Version documents and prefer the latest version at retrieval, while keeping history if users need to ask "what did the policy say in March?"
• Expose "last updated" dates in citations so users can judge recency themselves.
• Monitor ingestion lag per source and alert when a connector stops producing updates.

Changing the embedding model or chunking means re-embedding the corpus, so build a new index side by side and switch over after evaluation.

Cost and latency

A typical request spends time on identity resolution, query rewriting, parallel retrieval, reranking and generation. Generation usually dominates both cost and latency, and its cost scales with how much context you stuff into the prompt. Levers that work:

• Send fewer, better chunks. A good reranker lets you pass five chunks instead of twenty, cutting tokens and often improving answers.
• Stream responses so perceived latency is time to first token, not time to full answer.
• Route by difficulty. Smaller, cheaper models handle query rewriting, classification and many straightforward answers; reserve the largest model for complex synthesis.
• Use prompt caching where your provider supports it, for long, stable system prompts.

Set a p95 latency target and a cost-per-query ceiling, and track both per release. For budgeting the build itself, VOCSO publishes indicative ranges: a proof of concept at $12K–$20K over about six weeks, a focused production build at $20K–$60K, and a full production system at $60K–$150K+. Our AI development cost guide explains what drives a project toward each end.

Observability

When a user reports a bad answer, you need to replay exactly what happened. Trace every request with: the original and rewritten query, user identity and resolved groups, candidates from each retriever with scores, reranked results, the final prompt (or a reference to it), model and version, the answer, citations, latency per stage and token counts.

Pair traces with user feedback and review negative feedback weekly; recurring themes become new golden-set questions. Restrict access to traces, because they hold the same sensitive content as the source systems.

Common failure modes and fixes

RAG rarely lives alone. Most enterprise deployments also need connectors into CRMs, ticketing and data warehouses, which is where AI integration work comes in, and many end up as an internal assistant with role-aware behavior, the focus of our enterprise AI chatbot development practice.

FAQ

Do I need a dedicated vector database for enterprise RAG?

Not always. pgvector with Postgres full-text search handles hybrid retrieval for many corpora, and OpenSearch or Elasticsearch offer vector and keyword search in one engine. A dedicated vector database makes more sense at very large scale or for specific filtering and performance needs.

How do I handle document-level permissions in RAG?

Store each document's access control list as metadata on every chunk, resolve the user's identity and group memberships at query time, and apply them as a filter inside the retrieval query so restricted chunks never reach the model. Sync permission changes frequently and test for leaks with users of different roles.

What chunk size should I use for RAG?

There is no universal number. Start by splitting on document structure (sections and headings), cap chunk size to a few hundred tokens with modest overlap, and prepend the title and section path. Then tune using recall@K on your own golden set, because the best size depends on your documents and questions.

How do you evaluate a RAG system in production?

Evaluate retrieval and generation separately. Measure retrieval with recall@K and ranking metrics on a golden question set, and measure answers for faithfulness, correctness, citation accuracy and appropriate abstention. Run the suite on every change, and feed real user feedback back into the test set.

RAG or fine-tuning for company knowledge?

For knowledge that changes and must respect permissions, RAG is usually the right starting point: content updates without retraining and every answer can cite its source. Fine-tuning is better suited to teaching a model style, format or domain-specific behavior, and the two can be combined.

If you are planning a RAG system that has to pass a security review and hold up under real usage, our team can help with architecture, evaluation and the production build. See how we approach RAG development services and get in touch to talk through your corpus, access model and goals.