On-Premise RAG Assistant for the Company Wiki

The problem

Technical knowledge in a manufacturing company is real, extensive and almost unfindable. It lives in a wiki that grew organically, where the answer exists but the search box will not surface it.

An LLM is an obvious fit for that. Sending internal process documentation to a third-party API is not. So the constraint was fixed before the project started: everything runs on our own infrastructure.

Approach

Standard retrieval-augmented generation, executed with attention to the parts that actually determine whether it works.

   wiki pages ──→ chunking ──→ embeddings ──→ vector index
                                                   │
   user question ──→ embedding ──→ retrieve top-k ─┤
                                                   ▼
                              prompt = context + question ──→ local LLM

The interesting engineering is not the diagram — everyone has that diagram. It is:

• Chunking that respects document structure. Splitting on a fixed token count cuts tables and procedures in half and produces confidently wrong answers. Chunking on headings and keeping the section path as metadata fixed more retrieval failures than any embedding model change.
• Answers that cite their source. Every response points back to the wiki page it came from. This is what makes the system usable in a technical context: the user can verify, and the failure mode becomes “that page is outdated” rather than “the AI lied to me”.
• Refusing to answer. If retrieval returns nothing relevant, saying so is the correct output. Tuning that behaviour is unglamorous and it is the difference between a tool people trust and a toy.

Benchmarking LLMs on our own data

Public benchmarks tell you how a model performs on public tasks. They do not tell you how it handles our terminology, our document structure, our languages or our questions.

So I built a framework to benchmark candidate models against company-relevant prompts and data: a curated evaluation set drawn from real questions, scored consistently across models, with the retrieval component held fixed so the comparison isolates the generator.

That turned model selection from a matter of opinion and release-note hype into a measured decision — and it stays useful, because every new open-weight release can be run through the same harness.

What I would do differently

I would treat the wiki itself as part of the system from day one. Retrieval quality is bounded by document quality, and the project surfaced a lot of outdated and duplicated content. The chatbot is, among other things, an excellent documentation audit tool.