On this page
Why a language model does not know your price list
With RAG an AI assistant answers from your manuals, contracts and procedures instead of from memory. That memory is what a large language model learned in training: plenty of general knowledge, nothing about the terms you agreed with a supplier last month. Ask such a model about your own rates anyway and you get an answer that sounds right and is not. That is an AI hallucination.
Think of an open-book exam. The model stays the same, but for every question it gets the right pages in front of it. The idea comes from a paper by Patrick Lewis and eleven colleagues, with researchers at Facebook AI, first published in May 2020. In September 2020 Facebook AI spelled out what it means for a business: you change what the system knows by swapping the documents, without retraining the model. It has since become the standard way to put AI to work on company knowledge.
Five steps between a question and a sourced answer
- Chunking. Your documents are cut into passages of a few hundred tokens each, about a paragraph or half a page, each with its source.
- Embedding. An embedding model turns every passage into a list of numbers that captures its meaning; Voyage AI returns 1,024 by default. Passages on the same subject sit close together, across languages too.
- Storing. Text, source and numbers go into a database that searches by similarity.
- Retrieving. The question becomes a list of numbers as well, and the system pulls the passages closest to it. A second search on exact words catches part numbers and error codes that a meaning-based search can miss.
- Generating. The best three to ten passages go to the model with the question and an instruction to answer only from them and cite the source. How to write that instruction is in prompt.
Retrieval decides the quality, not the model. Anthropic measured that passages with a line of added context, plus a search on exact words, cut failed retrievals by 49%; with an extra ranking step (reranking), by 67%.
Index once, retrieve on every question
Once per document: indexing
- 2,000 pagesDocumentsManuals in Dutch, German and French, plus service reports.
- ChunkingPassagesA paragraph or half a page each, with the source attached.
- EmbeddingNumbersEach passage becomes a list of 1,024 numbers that captures its meaning.
- IndexDatabaseText, source and numbers together, searchable by similarity.
Every question: retrieve, then answer
Asked in German: “Fehler F28 an der Siegeleinheit, was jetzt?”
Service manual (FR), p. 140.86
The page on F28, written in French.
Service report, last visit0.79
Which machine this customer runs and what was replaced.
Operator manual (DE), p. 30.52
Returns procedure0.31
Only passages above the line go to the model
F28 means the sealing jaws do not reach temperature. Check the heater connector first; if the fault returns, replace the heating element, as on the last visit.
- Service manual (FR), p. 14
- Service report, last visit
The model stays the same. What changes is the text it gets to read for each question.
One manual set in three languages
Take a maker of packaging machines with 2,000 pages of manuals in German, French and English, plus service reports from customers across Europe. A field engineer types in German: "Error F28 on the sealing unit, what now?" The system finds the French page that describes F28 and the report from the last visit to that customer, and the answer names both sources. A multilingual embedding model makes that work: meaning lies close together even when the words differ.
When a new machine ships, you index its manual. Nobody retrains anything, which is the real difference with fine-tuning.
Retrieve, retrain or send everything
| Approach | What it does | Choose it when | What you pay |
|---|---|---|---|
| RAG | Finds the matching passages per question and passes only those to the model | You have more than a few hundred pages that keep changing | Indexing once, then a few thousand tokens per question |
| Everything in the prompt | Sends the whole knowledge base with every question | Your knowledge stays under about 500 pages, such as a price list and ten procedures | The full text per question, cheaper with prompt caching |
| Fine-tuning | Trains a model further on examples so it learns a style or format | You want a fixed tone or output, not new facts | A training run plus pricier use, again after every change |
| A plain chatbot | Answers from its training and whatever you paste in | You write general text without your own data; see ChatGPT vs Claude | A subscription per user |
Fine-tuning teaches a model how to write; RAG gives it what to know. Facts that change, such as prices, stock and rules, belong in RAG.
The bill per question: usually cents
For the machine maker above, in US dollars before tax:
| Item | How it is counted | Cost |
|---|---|---|
| Indexing 2,000 pages, once | about 800,000 tokens with voyage-4 at 0.06 dollars per million | 5 cents, and the first 200 million tokens are free |
| One question on Claude Sonnet 5.5 | 3,000 tokens in (question, instruction, five passages) and 400 out | about 1 cent |
| The same question on Claude Haiku 5.5 | same tokens at 0.10 and 0.50 dollars per million | about 0.05 cents |
| 1,000 questions a month | Sonnet 5.5 | about 10 dollars |
| Everything in the prompt, for comparison | 200,000 tokens per question from the Sonnet 5.5 cache | about 2.4 cents per question |
So the model is rarely the big line. The real work is cleaning documents, making scanned PDFs readable, setting permissions per employee and testing until the answers hold. Run your own volume through the AI cost calculator; what the build itself costs is in AI chatbot cost.
Where an assistant on your own documents goes wrong
- Scanned PDFs without textA scan is a picture; without text recognition there is nothing to find.
- Tables split in twoA price table cut across two passages gives half answers. Split on headings and keep tables whole.
- Old versions next to newWith the 2024 and the 2026 rates both indexed, the model sometimes picks the wrong one. Keep one source per subject.
- Everyone sees everythingAn assistant that also searches payslips must know who is asking. Permissions belong in retrieval, not in the prompt.
- Documents from outsideA supplier's PDF can carry hidden instructions; see prompt injection.
- Personal dataWhen a provider processes your documents you need a data processing agreement. The full roadmap is in implementing AI in your business.
Ready for a pilot? Bring these
- Fifty real questions your team asked last month, each with the right answer.
- The documents those answers live in, latest version, as text rather than scans.
- A list of who may see what: office, field staff, management.
- Which provider processes your data, and in which region.
- An agreement on what happens when the answer is not in the documents.
Questions buyers ask before a pilot
Do I need to train my own AI model?
Almost never. You use an existing model and hand it the right passages for each question. Indexing a new manual costs a few cents; retraining a model takes days and has to be repeated after every change.
From how many documents does this pay off?
Below roughly 500 pages you can send the full text with every question, especially with prompt caching. Above that, or when documents change often, retrieval is cheaper and faster.
Do my documents stay private?
That depends on where it runs. Pick an environment where your data is not used to train models, sign a data processing agreement with the provider, and set permissions per employee so an intern never sees payslips.
Does it work across languages?
Yes. A multilingual embedding model places meaning close together even when the words differ, so a question in German finds a French manual and the answer comes back in German.