“Can we have a ChatGPT for our company documents?” is one of the most common AI requests we hear. The technique behind a good answer is retrieval-augmented generation (RAG). This guide explains how it works, where it goes wrong and how to build it properly. To build one, see our AI agents and chatbots service.
How does a RAG chatbot work?
A RAG chatbot works in two phases: preparing your content once, then retrieving and answering for every question.
Preparation (ingestion):
- Collect documents from shared drives, SharePoint, wikis, help desks or databases.
- Extract clean text, including from PDFs, scans and tables.
- Chunk the text into passages small enough to be precise, large enough to keep meaning.
- Embed each chunk as a vector that captures its meaning, and store it in a vector index with its source and permissions.
Answering (retrieval and generation):
- The user’s question is embedded the same way.
- The most relevant chunks the user is allowed to see are retrieved, often combining meaning-based and keyword search.
- The model receives the question and those chunks, with instructions to answer only from them and cite sources.
- If nothing relevant is found, the chatbot says so and offers a hand-off.
When is RAG the right approach?
| Situation | RAG fit | Why |
|---|---|---|
| Policies, manuals, SOPs, contracts | Strong | Answers exist in text and change over time |
| Product documentation and FAQs | Strong | Precise, citable answers |
| Live figures (stock, orders, balances) | Combine with tools | Query systems directly instead of documents |
| Analytical questions across many records | Use “ask your data” | Database queries suit aggregation better |
| Highly creative or open-ended tasks | Weak | Grounding is less relevant |
Most business assistants combine RAG for documents with tool calls for live data, as described in how to add AI to your existing software.
What usually goes wrong?
- Poor extraction: tables, headers and scanned pages turned into garbled text. Fix: proper parsing and OCR, checked on samples.
- Bad chunking: passages cut mid-sentence or separated from their headings. Fix: structure-aware chunking that keeps titles and sections.
- Outdated content: old versions of policies answered alongside new ones. Fix: a single source per document, automatic re-indexing and version metadata.
- Ignored permissions: a user receives an answer from a document they cannot open. Fix: permissions stored with every chunk and enforced at retrieval.
- Confident guessing: the model fills gaps from general knowledge. Fix: strict grounding instructions, citations and refusals.
- No measurement: nobody knows if accuracy is improving. Fix: an evaluation set of real questions, scored on every change.
How do you keep a RAG chatbot accurate over time?
Treat it like any production system. Keep an owner for the content, re-index automatically when documents change, review a sample of live conversations each week, add failed questions to the evaluation set, and re-run evaluation before every change to prompts, models or chunking.
What does it cost to run?
Running cost includes hosting for the application and vector index, model usage per question (which depends on how many passages are sent and the model used), re-indexing when documents change, and ongoing evaluation and support. The full list of drivers is in what drives AI agent development cost.
Questions to ask before you start
- Which questions should the chatbot answer, and for whom?
- Where do the source documents live, and who owns them?
- How often do documents change, and how are old versions handled?
- Who may see which documents?
- What should happen when the answer is not in the documents?
- Which channels will people use: web, Teams, WhatsApp, inside an app?
- What data may be sent to a cloud model, and what must stay in-house?
- How will we measure accuracy before and after launch?
Get a scoped estimate
Every system is different. Send us your requirements and we’ll come back with a module-by-module scope, timeline and infrastructure plan. We reply within 1 business day.
Book a scoping call or message an engineer on WhatsApp, and see how we build AI agents and chatbots.