On-premise AI for internal documents: what it takes to run a Vietnamese LLM on your own servers

The question most organizations are actually asking is this: can our staff query internal documents in plain language without those documents ever leaving our control? The short answer is yes. On-premise AI means the language model and the entire data path sit on servers you own. Documents are read, chunked, embedded and answered locally. No packet carrying document text ever leaves your network.

The longer answer is more involved, and that is what this article is about. Getting a Vietnamese LLM running on your own server is the easy part. The hard parts are the vector store, the permission layer, the document refresh process, and being honest about what a locally hosted model can and cannot do compared with a large cloud service.

Why organizations hesitate to send internal documents to cloud AI

The hesitation is not vague anxiety. It has four specific sources.

• Data terms are hard to read and change often. The same vendor typically runs several service tiers with different policies: free, personal paid, enterprise, and API. Commitments not to train on your data usually apply only to some tiers. Meanwhile, staff sign up with personal accounts and paste contracts into them every day in every organization, entirely outside whatever your legal team signed.
• Retraining and incidental retention. Even where a vendor does not train on your content, that content may be retained for abuse monitoring, debugging, or to satisfy legal demands in the vendor's jurisdiction. Retention periods and internal access scope are things you cannot verify from the outside.
• Personal data protection duties. If internal documents contain personal data, pushing them to an offshore service is a cross-border data transfer, with the corresponding impact assessment file required under the Personal Data Protection Law 91/2025/QH15 and Decree 356/2025/ND-CP. This is real paperwork, not a hypothetical risk. We covered the trade-off in more depth in on-premise versus cloud from a compliance angle.
• Sector-sensitive data. Medical records, credit files, engineering drawings, unopened tender documents, anything classified under sector rules. For this category the question is not whether the vendor is trustworthy, but whether you are permitted to send it out at all. Often the answer is simply no.

There is one more layer. AI Law 134/2025 imposes transparency and risk-management duties on AI systems. A system running on your own infrastructure is far easier to document, log and defend than a rented black box.

What on-premise AI is actually made of

Many people picture local AI as installing a chatbot on a server. In practice it is five components, and the language model is only one of them.

• The language model. The part that produces the answer. The most discussed component, and the one that least determines quality.
• The embedding model. The part that turns a passage of text into a numeric vector. Embedding quality on Vietnamese decides whether the system finds the right passage. Get this wrong and even an excellent language model is just writing fluently about the wrong document.
• The vector store. Where vectors live alongside metadata for each chunk. It must retain provenance: which document, which department, which sensitivity level. Skipping that metadata is the root cause of every cross-department leak described below.
• The permission layer. A filter that runs before retrieval, not after. Only chunks the asker is entitled to read may enter the context.
• The question-answering interface. Where users type and receive answers with source citations. Without citations, do not deploy, because users have no way to verify anything.

What RAG is

RAG stands for retrieval-augmented generation. The mechanism is simple: when a user asks a question, the system does not query the model directly. It first searches the document store for the most relevant passages, then hands those passages plus the question to the model and asks it to answer strictly from what it was given.

Why this matters: the model never has to be trained on your documents. No fine-tuning, no retraining every time a new policy is issued. Add a new regulation to the store, reindex a few pages, and the system answers correctly immediately. This is also why RAG suits constantly changing internal documents, while fine-tuning suits teaching a model a fixed writing style or output format.

The security consequence deserves attention too. Because documents live in the context of a single query rather than in the model weights, removing a document from the vector store removes it from every subsequent answer. Fine-tuning offers no equivalent undo button.

Open-source Vietnamese LLMs you can run on your own servers

As of this writing there are three pragmatic paths.

• Vietnamese-specific models. PhoGPT from VinAI Research is an open-source Vietnamese generative series comprising the base PhoGPT-4B and the chat variant PhoGPT-4B-Chat, pre-trained from scratch on a Vietnamese corpus with an 8192-token context length. It reads Vietnamese naturally and is light to host. Its limits are narrower background knowledge and weaker multi-step reasoning than larger models.
• Multilingual models that include Vietnamese. SeaLLMs from DAMO NLP Singapore targets Southeast Asian languages including Vietnamese and ships open weights. Beyond that, mainstream open model families from a few billion to a few tens of billions of parameters all handle Vietnamese well enough for summarization and document question answering.
• Vision models for documents. For scanned paperwork you need a model that reads images, not just text. The Vintern family from 5CD-AI is an open model line aimed at Vietnamese OCR and document understanding, and it runs on-premise. This is the commonly forgotten piece: a great deal of internal archive material is photographs of paper, not text with a character layer.

A note on evaluation. Do not pick a model from a general leaderboard. Take roughly fifty real questions your staff actually ask, run them through a few candidate models against your own document store, and have people who know the business grade the answers. Public leaderboards measure tasks that are not looking up your internal policy.

How much GPU you actually need

This deserves plain speaking, because wrong expectations here sink projects.

The basic arithmetic: GPU memory for model weights is roughly parameter count times bytes per parameter. That is two bytes at 16-bit precision, one byte at 8-bit quantization, half a byte at 4-bit. On top of weights you need room for the context cache, which grows with input length and the number of concurrent users. Three tiers follow from that.

• Single-department pilot. One server with one mid-range GPU is enough for a few-billion-parameter model in quantized form serving a handful of concurrent users. Enough to answer whether the system has real value before spending real money.
• Organization-wide, steady load. A dedicated large-memory GPU server running a mid-sized model at higher precision for dozens of concurrent users. This is where most Vietnamese enterprises land.
• Thousands of users or genuinely large models. A multi-GPU cluster, with its own networking, cooling and power design. Costs step up sharply at this threshold. Sovereign AI infrastructure in-country now offers options here, but it is not a sensible starting point for anyone.

State the limit clearly to leadership: an open model on one server in your own machine room will not match the largest cloud models. It does not need to. For finding and summarizing a policy passage that retrieval has already located, the capability gap narrows considerably compared with open-ended reasoning. Selling accurate expectations upfront matters more than selling an impressive number.

Access control is where cross-department leaks happen

The most common incident with internal AI is not an external attacker. It is an employee asking an ordinary question and receiving another department's content.

The failure mode is easy to picture. The implementation team loads the entire shared drive into the vector store to move fast, planning to filter permissions later. But the answer has already been generated from a context containing that passage. Filtering at the output is filtering too late. Three rules are non-negotiable.

• Filter before retrieval. Every vector store query must carry the asker's permission scope. No out-of-scope chunk enters the context, not even one sentence.
• Inherit permissions from the source system. Read rights must come from the document management system, never from a parallel permission table. A parallel table drifts within months, and then nobody knows which one is correct.
• Log questions and the sources used. Every answer should record which documents fed it. When a leak is suspected, this is the only thing that lets you reconstruct events.

One detail teams miss: when someone transfers department or leaves, permissions must change in the vector store, not only on the source folder. Any divergence between the two is a hole.

In Tetra eOffice, the lookup assistant uses the same permission set as the documents themselves, so users can only ask about material they were already cleared to read. More on the Tetra eOffice page.

When you do not need local AI

Not every problem justifies building a system.

• A small document set. A few hundred documents plus a good full-text search will cover most of the need, faster and far cheaper.
• Data that is already public. If the material is legislation, marketing collateral or content already published on your portal, building dedicated infrastructure to protect it is cost without value.
• The problem is really template extraction. If the need is pulling fields from thousands of identically formatted forms, that is not a question-answering problem. A template-based digitization and extraction system such as Molly fits better and produces far more consistent results than asking a conversational model. We wrote about that path separately in digitizing old paperwork with on-premise AI.
• No one to operate it. If nobody in the organization can watch GPU health, update models and handle wrong answers, solve the staffing problem first.

A pilot-first roadmap

The lowest-risk path is incremental, with an explicit stopping criterion at each step.

• Pick one department and one document type. HR with the internal policy set, or legal with the contract template library. A narrow scope makes honest evaluation possible.
• Write the benchmark questions before building anything. Around fifty real questions with correct answers drafted by people who know the business. This is the only measurement worth trusting.
• Run the pilot on the smallest acceptable hardware. The goal of this phase is proving value, not proving throughput.
• Measure the share of correct answers that cite a source. An answer that sounds good but cannot point to a source counts as wrong.
• Test the permission layer by trying to break it. Give a low-privilege account questions aimed squarely at restricted documents. Nothing leaks, or you do not expand.
• Expand by department, not by document count. Each new department brings its own permission rules that need rechecking.

Each step should have a deadline measured in weeks, not quarters. If after six to eight weeks the first department's users are not coming back on their own, the problem is the use case rather than the model, and scaling up will not rescue it.

Conclusion

On-premise AI is not a weaker version of cloud AI. It is a different choice about where data gets processed, and for internal documents containing personal data or sector-sensitive information it is often the only permissible choice. Picking the model is the easy part. What decides success is retrieval quality, the permission layer, and honesty about expectations.

If your organization is weighing this path and wants to talk specifics about your document store, permissions and right-sized hardware, book a consultation with the Tetra team.