RAG and knowledge bases for AI agents: a simple guide

RAG stands for Retrieval-Augmented Generation. It allows an AI system to retrieve relevant information from a defined knowledge source and use it while generating an answer. This is useful when an agent needs current, internal or specialised information that may not be present in the model’s original training.

RAG does not normally retrain the language model. Relevant passages are supplied at the time of the request, which allows an organisation to update its documents without building a new model.

How RAG works

First, approved documents such as policies, product manuals or procedures are collected. They are usually divided into smaller chunks. Each chunk is represented in a form that supports semantic search; embeddings are a common method.

When a user asks a question, the system creates a searchable representation of the query. It retrieves the most relevant chunks and sends them to the language model with the question. The model writes an answer using this context. A good implementation identifies its sources and says when evidence is insufficient.

Example: an internal process guide

An employee asks how a purchase above a certain threshold should be approved. The retrieval system finds current sections covering limits, roles and documentation. The agent summarises the steps and links to the relevant policy version.

If the policy does not cover an exception, the agent should not invent a rule. It should explain that the source is insufficient and refer the employee to the responsible team. That behaviour must be instructed and tested.

RAG compared with uploaded files

A project containing a few uploaded documents may be enough for a small solution. A dedicated retrieval architecture is more relevant when material is larger, changes frequently or serves several applications. There is no fixed boundary, and many modern project tools use retrieval behind the scenes.

RAG is not the same as MCP. RAG describes how relevant knowledge is retrieved for an answer. MCP is a protocol for connecting an AI application to resources and tools. An MCP connection could expose a search service that uses RAG.

What determines quality?

Source material is crucial. Old, conflicting or poorly labelled documents create uncertain answers. Documents need owners, versions, effective dates and retirement rules. Chunking also matters: fragments that are too small lose context, while very large fragments add noise.

Test retrieval separately from the final answer. If the correct passage is never retrieved, the model cannot reliably use it. Measure both retrieval quality—whether the right source was found—and answer quality—whether it was interpreted correctly.

Common failures

A citation does not automatically prove that a claim is supported. Check whether the cited passage actually matches the answer. Another mistake is loading an entire document archive without curation. More data is not always better.

Documents may also contain instructions that attempt to manipulate the agent. Treat retrieved content as data rather than higher-priority instructions. Limit actions that can follow from untrusted text.

When does RAG fit?

RAG is useful when answers must rely on a bounded, changing body of knowledge. It is less useful for mainly creative tasks, where reliable sources do not exist or where a structured API can answer exact database questions more efficiently.

In summary

RAG retrieves relevant knowledge before a model produces an answer. It can improve currency and traceability, but only when sources, search and testing are well managed. A knowledge base is an ongoing service, not a one-time upload.

Sources

• Anthropic: RAG for Projects
• Microsoft Learn: Retrieval-augmented generation
• NIST: AI Risk Management Framework

Last reviewed: 16 September 2026.