Deploying on-premise RAG for internal documents: a practical guide

"Let AI read our internal documents and answer staff questions" — it sounds simple, and the demo always looks great. But in real operation, most projects break on things no demo mentions: a dirty document store, loose permissions, and a general feeling that "it seems fine" standing in for actual numbers. This is a practical guide to deploying RAG over internal documents on your own infrastructure, written for the person about to build it rather than the person watching a product pitch.

What RAG is, briefly

RAG (Retrieval-Augmented Generation) is not retraining a model. The idea is much simpler: when a question arrives, the system finds the most relevant document passages, puts them into context, and lets the language model answer based on exactly those passages, with source citations.

The crucial point is that the model does not need to know your documents in advance. They are never baked into its weights. It reads only the passages handed to it at query time. So when you update a document the system knows immediately, with no retraining; remove a document from the store and it vanishes from every subsequent answer. Answers carry sources users can verify, and that is the biggest difference from a chatbot that responds fluently but cannot point to where its claim came from.

Draw one boundary early: RAG is not fine-tuning. Fine-tuning teaches a model a tone, an output format, a fixed way of behaving; RAG supplies the model with fresh facts at the moment of the question. For internal documents that change constantly, RAG is almost always the right direction.

Why run it on-premise

Internal documents concentrate an organization's most sensitive material: contracts, HR records, operational documents, unopened bid files. Pushing them through a foreign cloud API means data leaves your control — bumping into personal-data protection duties and, for many organizations, simply not permitted. Running the model locally keeps the whole processing loop in-house: read, chunk, embed, retrieve, generate — no packet carrying document text ever leaves your network.

On-premise also gives you three things a rented service rarely can. Control over the data lifecycle: you decide what is stored, for how long, and who touches it. Accountability: every question and every source used sits in your own logs, ready for internal audit and for the transparency obligations that increasingly attach to AI systems. And stability: the model does not change behind your back after a vendor update, so today's behavior matches next month's. We covered this foundation in depth in on-premise AI for internal documents; this article focuses on building the RAG pipeline itself.

The architecture: how data flows end to end

Picture a straight pipeline that splits into two paths. The ingestion path (offline, run whenever new documents arrive): raw documents → cleaning and text extraction → chunking → embedding → storage in the vector store with metadata. The query path (online, run every turn): question → permission filter → retrieval of relevant passages → re-ranking → assembly into context with the question → local language model generates a cited answer. The two paths meet at the vector store. Hold that diagram in mind and each step below lands in its proper place.

1. Data preparation — the decisive step, and the most overlooked

This is where most projects fail, and almost no demo mentions it. If your documents are scans, photos and messy PDFs, then before talking about RAG you must digitize and clean them into readable data. Garbage in, garbage out — no model rescues an unreadable archive. Tables flattened into one line, headings fused to footers, OCR that mangles Vietnamese diacritics: each of these later becomes a wrong answer you can barely trace back.

For legacy paperwork this is its own problem: digitizing and extracting into structured data. That is what Molly does — and to be clear, Molly is not a RAG system or chatbot: it is a digitization, extraction and lookup layer aimed at large volumes of same-template documents. But a clean data store is exactly the raw material any RAG system needs. Do not conflate the two: RAG is the question-answering architecture; template-based digitization is the stage that comes before it.

2. Chunking

Long documents must be split into passages small enough for precise retrieval. Too large and one passage carries several topics, so retrieval returns noise; too small and you lose context, and the model reads a fragment dangling in mid-air. Splitting along natural structure — sections, articles, clauses, headings — usually beats a hard character count, because it respects meaning boundaries. Let passages overlap slightly so you never cut a key sentence in half, and keep the section heading on each passage so the model knows where the text belongs.

3. Embeddings and a vector store

Each passage becomes a numeric vector representing its meaning, stored in a vector store. The user's question also becomes a vector, and the system finds the semantically nearest passages. The quality of the embedding model on your language decides whether the system finds the right passage at all — get this step wrong and the finest language model is just writing prose over the wrong document. The vector store must hold metadata for each passage: which document, which department, which sensitivity level, which version. Skipping that metadata plants the seed for every cross-boundary leak discussed below.

4. Retrieval and ranking

Pull candidate passages, then re-rank to pick what is genuinely relevant. Combining semantic search with keyword search — a hybrid approach — usually beats either alone, especially for domain jargon, document numbers and contract codes, the kind of exact tokens semantic search tends to miss and keyword search catches precisely. A re-ranking step over the candidate set pushes the right passage to the top and trims how many passages enter the context, improving accuracy and cutting generation cost at once.

5. Generating answers with citations

A locally run language model answers based only on the retrieved passages, with citations to source documents. Open models such as Llama, Qwen and Gemma all run on-premise, including in a fully air-gapped environment. Requiring citations is not decoration: it is the mechanism that lets users verify and catch the model inventing things. Give the model one hard rule — if the context is not enough to answer, say "not found in the documents" instead of guessing — and you cut most confident hallucination at the source.

6. Access control — rarely discussed, easily disastrous

This is the most dangerous point. If your RAG system retrieves across the whole archive without checking the asker's permissions, you have built a machine that leaks data across boundaries with an official veneer: an employee asks something harmless and receives content from documents they may not read. The failure mode is easy to picture: the delivery team loads the whole shared drive for speed, meaning to filter permissions at display time. But the answer has already been generated from a context containing that passage — filtering at the output is filtering too late.

The principle: permissions must be applied at retrieval, not filtered at display. Which user may see which document — the system must know before it searches, and every query into the vector store must carry the asker's permission scope. Permissions should be inherited from the source document-management system, not maintained in a parallel table that drifts out of sync within months. And when a person changes department or leaves, permissions must change on the vector store immediately, not only on the source drive.

7. Evaluation — don't trust vibes

Build a set of your organization's real questions with correct answers, then measure: accuracy rate, hallucination rate, whether citations point to the right source, response time. An answer that sounds good but cannot cite a source counts as wrong. Without measurement you cannot tell whether each tweak made it better or worse — and RAG has many knobs to tweak: chunk size, embedding model, number of passages retrieved, ranking method, the instruction given to the model. A fixed question set turns gut-feel fiddling into a before-and-after comparison of numbers.

On-premise and air-gapped: what to plan for

Running in-house does not automatically mean fully air-gapped. If your organization truly needs an air gap, plan for three things: pre-load every model weight — language, embedding, re-ranking — inside the network perimeter, because there will be no downloads at runtime; establish a controlled file-transfer process for model and library updates rather than a pip install straight to the internet; and accept that all patching, monitoring and backup must be run by you. In return you get a system where no fact leaves the server room — exactly what many industry confidentiality regimes demand.

How much hardware is enough

Wrong expectations here sink whole projects, so let us be blunt. GPU memory for the weights is roughly the parameter count times the bytes per parameter: 2 bytes at 16-bit, 1 byte at 8-bit quantization, half a byte at 4-bit. Beyond the weights you need room for the context cache, which grows with input length and concurrent users. Three tiers follow: a mid-range GPU running a few-billion-parameter quantized model is enough for a single-department pilot; a large-memory GPU server running a mid-size model for dozens of concurrent users is where most enterprises settle; a multi-GPU cluster is needed only at the thousands-of-users scale, and cost jumps sharply at that threshold. Do not forget that embeddings and the vector store also need their own resources, usually on CPU and RAM, separate from the generation GPU.

State the limit plainly to leadership: an open model on one server in your room will not be as strong as the largest cloud model. It does not need to be. For finding and summarizing a passage that has already been retrieved, the capability gap narrows dramatically compared with open-ended reasoning.

Governance: permissions, logging, lifecycle

Permission-at-retrieval is the foundation; on top of it you need an operational governance layer. Log every question and the sources used, so you can reconstruct events when a leak is suspected. Tie the document lifecycle to the vector store: expired regulations, superseded versions and withdrawn files must be removed or flagged so they never surface in an answer. Assign clear ownership for monitoring quality and handling wrong answers — without that role, the system quietly degrades. Seen through the lens of sovereign AI for Vietnamese enterprises, governance sits inside a wider set of infrastructure and compliance choices.

Build vs. buy

There is no single right answer, but there is a right way to ask. Building yourself gives maximum control and no vendor lock-in, at the cost of feeding a team that understands the model, retrieval and permissions all at once, and owning every wrong answer the system produces. Buying a packaged solution shortens time-to-value and pushes the hard operations onto a vendor, at the cost of recurring fees and limited customization. Most organizations should start with a small pilot to understand their own problem before committing to either path — many teams discover that what they needed was a decent search box, and that is worth learning early.

Rollout checklist

• The problem: Confirm this really is natural-language question answering that needs synthesis across documents, not keyword search or template extraction.
• The data: Digitize and clean the documents, and check extracted-text quality before ingesting.
• A benchmark set: Write a few dozen real questions with correct answers, before building the system.
• Permissions: Apply access control at retrieval, inherit it from the source system, and test for escalation with a low-privilege account.
• Citations: Every answer must cite a source; an answer without one counts as wrong.
• Logging: Record the question and the sources used for each turn.
• Measurement: Run the benchmark set before and after each change, comparing numbers rather than impressions.
• An operator: Have someone responsible for watching the GPU, updating the model and handling wrong answers.

Not every problem needs RAG

Few vendors say this. Ask honestly: do you need natural-language question answering, or just to find the right document / pull a few data fields?

• Need to find documents by keyword → full-text search is simpler, faster, and doesn't hallucinate.
• Need structured fields (contract number, date, name) out of stacks of same-template paperwork → data extraction is far more accurate and verifiable.
• Only when questions are open-ended and require synthesis across documents does RAG truly earn its keep.

Picking the right tool for the problem saves far more than building a RAG system and discovering you needed a decent search box.

Conclusion

On-premise RAG is a solvable engineering problem, but the hard part isn't the model — it's clean data, correct permissions applied at retrieval, and honest measurement against a benchmark set. Organizations that take those three seriously get a lookup assistant people actually use; those that skip them get a pretty demo and a leak risk.

If you want to assess your internal-document problem and choose the right approach, book a consultation.