Enterprise RAG: AI Answers From Your Company Documents

Retrieval-augmented generation (RAG) is the technique that lets a large language model answer from your company's own documents: before replying, the system searches your manuals, contracts or tickets for the relevant passages and hands them to the model together with the question. Done well, the assistant cites a source for every answer and only shows each employee what they are allowed to see. This guide explains how it works, where it usually breaks and what your team needs to put it into production.

What is enterprise RAG, and how is it different from using ChatGPT?

Enterprise RAG is a system that connects a language model to an organization's internal sources so that its answers are grounded in verifiable documents rather than only in what the model learned during training. The approach was formalized in a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research, and it is now the most common architecture behind internal assistants, help desks and enterprise search.

The difference from pasting a PDF into a public chatbot is operational: a RAG system works across thousands of documents at once, stays current as sources change, logs who asked what, enforces per-user permissions and returns citations anyone can check. It is also the foundation of most business AI agents: before an agent can act, it has to find the right information.

How RAG works, step by step

A RAG system has two phases: preparing the documents (in the background) and answering each question (in seconds).

Ingestion: connectors pull documents from where they live (file shares, SharePoint, Google Drive, the ticketing system, databases) and extract the text. Scanned PDFs go through OCR first to get usable text.
Chunking: each document is split into passages, ideally following its structure: headings, sections, policy clauses, table rows. Every chunk keeps metadata such as source, page, date, department and permissions.
Embeddings: a model turns each chunk into a numeric vector that represents its meaning, so passages that say the same thing in different words end up close together.
Indexing: vectors and metadata are stored in a vector database or a search engine with vector support.
Vector search: the user question is embedded the same way, and the closest chunks by meaning are retrieved.
Hybrid search: semantic search struggles with SKUs, contract numbers and acronyms, so it is combined with keyword search (for example BM25) and the two rankings are fused.
Reranking: a specialized model reorders the top 50 to 100 candidates and keeps only the most useful passages for this specific question.
Generation with citations: the LLM receives the question plus the selected passages, with instructions to answer only from that evidence, cite each claim and say "I don't know" when the evidence is not enough.

Each step compounds. In a 2024 experiment, Anthropic reported that adding context to chunks and combining embeddings with BM25 cut retrieval failures by 49%, and adding reranking brought the reduction to 67%. Those numbers come from their test set and are not a guarantee, but they show that RAG quality depends more on retrieval than on the model that writes the answer.

Document permissions: the differentiator most implementations miss

Not everyone in a company can see everything: payroll, customer contracts, board minutes and HR files are restricted. A RAG assistant that ignores those permissions becomes the fastest way to leak internal information.

The rule is simple: enforce permissions at retrieval time, not in the prompt. Telling the model "do not reveal confidential data" is not a security control; a clever user or a poisoned document can get around it. The correct design:

Store on every chunk the list of users or groups allowed to read the source document (its ACL).
Authenticate users with your existing identity provider (Microsoft Entra ID, Okta, Google Workspace) and filter the search by their groups before anything is retrieved. Enterprise search engines call this security trimming.
Sync permission changes as often as content changes: when someone moves departments, they should stop seeing the old team's files in the assistant too.
Partition indexes per customer or business unit in multi-tenant setups to prevent cross-context leaks.
Log which chunks were retrieved for every answer so you can audit access later. That evidence also supports SOC 2 audits and, in healthcare, the HIPAA minimum necessary standard.

OWASP lists this risk in its Top 10 for LLM Applications (Vector and Embedding Weaknesses: LLM09 in the 2026 edition, LLM08 in 2025) and recommends permission-aware vector stores and strict data partitioning.

Keeping knowledge fresh

An assistant that quotes last year's PTO policy loses the team's trust within a week. To prevent it:

Incremental sync: re-index only what changed, hourly or daily depending on the source, instead of reprocessing everything.
Real deletion: when a document is deleted or archived at the source, its chunks must leave the index.
Visible dates and versions: citations should show the document date so users can spot stale content.
Content owners: each collection has an owner who reviews unanswered questions and outdated documents.
Prefer the current source: when two versions of a procedure exist, metadata lets retrieval favor the latest approved one.

Typical sources and what to watch for

Source	Examples	Watch for
PDFs and office files	Handbooks, SOPs, contracts, spec sheets	Scanned PDFs with no text layer, tables that break on extraction, duplicate versions
SharePoint, OneDrive, Google Drive	Department folders, policies, decks	Inherited folder permissions and "anyone with the link" sharing
Support tickets and email	Resolved cases, customer replies	Customer PII, wrong answers that became tribal knowledge
Databases and ERP/CRM	Inventory, order status, balances	Fast-changing data is better queried live through tools or APIs than indexed
Wikis and knowledge bases	Confluence, Notion, intranet	Abandoned pages with no owner or review date

For live systems like your ERP or CRM, the open Model Context Protocol (MCP) is increasingly used to let the assistant query data with the user's own permissions.

The most common quality pitfalls

Naive chunking: splitting every 500 words regardless of structure separates a rule from its exception, or a table from its header. The model gets half a paragraph and answers wrongly with full confidence.
Stale or conflicting documents: RAG does not clean your repository; if three versions of a policy exist, the assistant can cite any of them.
No citations: without a source nobody can verify, and errors go unnoticed. Citations should link to the document and, ideally, the page.
Hallucinations when evidence is missing: when retrieval finds nothing useful, models tend to fill the gap. An explicit instruction to abstain, plus test cases for it, is mandatory.
Too much context: sending 40 chunks "just in case" raises cost and degrades answers; models make worse use of information buried in the middle of a long context.
No measurement: without a set of real questions with expected answers, every change is a guess. See our guide on evaluating AI assistants.

Security and privacy

Beyond permissions, RAG introduces its own attack surface. The biggest one is indirect prompt injection: an indexed document or email can carry hidden text that tries to give the model instructions. We cover defenses in our prompt injection guide, and the full risk map is in our OWASP Top 10 for LLMs guide.

Practical controls for US companies:

Treat the vector index as a copy of your documents: encrypt it, restrict admin access and include it in your data inventory and retention policy.
Review your LLM provider's data retention and training terms, and prefer enterprise agreements that exclude your data from training.
Keep regulated data (PHI, financial records, data covered by state privacy laws) out of the index unless the use case requires it and the vendor contracts allow it, such as a BAA for HIPAA.
Map the system to the NIST AI Risk Management Framework and its Generative AI Profile if you need a defensible governance story for auditors or customers.

Project phases and what your company must provide

Discovery (about 1 week): use cases, users, sources and permission rules, plus 50 to 100 real questions collected from the team.
Prototype on real data (2 to 4 weeks): one or two sources ingested, hybrid search, citations and a first quality baseline. Through our AI development services this phase starts at USD 4,000.
Hardening: SSO integration, synced permissions, logging, security testing and a larger evaluation set.
Gradual rollout: one pilot department, thumbs-up/down feedback, weekly tuning.
Operations: monitoring quality, cost and stale content.

What your company provides: read access to the sources, a business owner with a few hours a week, real questions with correct answers, the rules for who can see what and, above all, the decision about which documents are the current source of truth. As a nearshore team in Quito working in UTC-5, we overlap fully with US Eastern and Central business hours for those weekly reviews. Total cost depends on the number of sources and users; we break it down in the cost of implementing AI in business.

Frequently asked questions

Does RAG eliminate AI hallucinations?

No. RAG greatly reduces made-up answers because the model works from specific evidence, but if retrieval returns the wrong or incomplete passages, the model can still answer incorrectly. A solid system requires citations, abstains when evidence is insufficient and is tested against a set of real questions before every change. Hallucinations are managed through design and continuous evaluation; they do not disappear on their own.

Are my documents used to train the model?

In a RAG system your documents are not used to train the model. They live in an index you control, and only the relevant passages are sent to the model at answer time. You should still review your LLM provider's terms on data retention and training use, sign an enterprise agreement where possible, and confirm the setup fits your obligations under HIPAA, state privacy laws or customer contracts.

How many documents do I need to start with RAG?

There is no minimum. A useful assistant can start with a single 50-page operations manual or with thousands of support tickets. What matters is that the documents are current, the use case is specific and you have a set of real questions to measure against. Starting with one clean, well-evaluated source beats indexing the entire company drive without curation.

Can a RAG assistant respect SharePoint or Google Drive permissions?

Yes, if it is designed for it. Each indexed chunk stores the users or groups allowed to read the original document, and every query is filtered by the signed-in user's identity before any content is retrieved. Permissions must be synced with the source on a schedule. What does not work is relying on instructions to the model to hide information: the filter has to live in the search layer.

Should I use RAG or fine-tuning so the AI knows my documents?

For knowledge that changes and needs citations, RAG is almost always the right choice: it updates instantly, lets users verify sources and supports permissions. Fine-tuning adjusts a model's behavior, format or style; it is not a reliable way to make it memorize documents. We compare both in detail in our RAG vs fine-tuning guide.

Conclusion

A working enterprise RAG system is defined less by the model it uses than by everything around it: structure-aware chunking, hybrid search with reranking, verifiable citations, permissions enforced at retrieval time and a process that keeps knowledge current. Start with one specific use case, one clean source and real questions to measure against, and within a few weeks you will know whether the assistant delivers value before you scale it.

If you want an assistant that answers from your company documents and respects who can see what, explore our AI development services or book a free consultation to review your sources with our team.

Enterprise RAG: How to Build an AI Assistant That Answers From Your Company Documents