MCP vs RAG: When to Use Each

RAG and MCP answer different questions. RAG (retrieval-augmented generation) is a technique: find the most relevant passages in a knowledge base and add them to the prompt. MCP (Model Context Protocol) is a protocol: a standard way for an AI application to connect to tools, data and prompts. Use RAG when the problem is "the model does not know my documents." Use MCP when the problem is "the model must reach and act on my systems." Often you use both. Checked 31 October 2026.

The short version

What each one is

RAG

The term comes from a 2020 paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," by Patrick Lewis and colleagues. The abstract frames the problem: language models store knowledge in their parameters, but their ability to access and manipulate it precisely is limited, and providing provenance for decisions and updating knowledge are open problems. RAG combines a generator with non-parametric memory, an external store you can search.

Anthropic's description of the common practical version is simpler. RAG "retrieves relevant information from a knowledge base and appends it to the user's prompt." The usual pipeline:

1. Break the knowledge base into chunks, usually no more than a few hundred tokens.
2. Use an embedding model to turn each chunk into a vector that encodes meaning.
3. Store the vectors in a vector database that searches by semantic similarity.
4. At query time, embed the question, fetch the closest chunks and add them to the prompt.

MCP

The MCP site calls it "an open-source standard for connecting AI applications to external systems": data sources such as local files and databases, tools such as search engines and calculators, and workflows such as specialized prompts. An MCP host (an AI application) keeps one client per server over JSON-RPC 2.0, with stdio for local servers and Streamable HTTP for remote ones. Servers offer tools, resources and prompts, and can ask the user for information (elicitation).

RAG is something you build inside a pipeline. MCP is something your application speaks.

Side-by-side facts

Where they overlap

The confusion comes from MCP's resources. The spec lists resources as "context and data, for the user or the AI model to use," which sounds like retrieval. And an MCP tool can run a search and return text, which is retrieval by another route.

The difference is what each is for. RAG is a way to decide which text goes into the prompt, using similarity over an index you built. MCP is the connection layer. A search tool on an MCP server may use RAG inside. Or it may run a SQL query, call a vendor's search API, or read a file, none of which involve embeddings.

So the two sit at different layers. RAG is an answer to "how do I find the right passages." MCP is an answer to "how does the agent reach the system that holds them."

When you do not need RAG

Anthropic's contextual retrieval post says it plainly: "Sometimes the simplest solution is the best. If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt," with no RAG needed. With prompt caching, it says, costs fall by up to 90 percent and latency by more than 2x.

Today's models list context windows near one million tokens, so the size at which you can skip retrieval keeps growing. A small handbook, a product manual or a single repository may fit. When it fits, you remove the chunker, the embeddings, the vector database and a whole class of retrieval errors.

You still pay for those tokens on every call unless you cache the prefix. Test the cost on your own traffic.

When RAG earns its place

• The knowledge base is larger than what you want to send on each call.
• Questions touch a small part of a large corpus.
• You need to cite the specific passage behind an answer.
• The data changes, and you can re-index on a schedule.

Retrieval has failure modes

Anthropic's post is candid about them. Traditional RAG "removes context when encoding information," so a chunk that says "the company's revenue grew by 3%" does not say which company or quarter, and retrieval can miss it. Embeddings capture meaning but can miss exact matches: its example is a query for "Error code TS-999," where an embedding model may return general error-code content and miss the exact string. A lexical method, BM25, finds exact terms.

Its fix, Contextual Retrieval, adds chunk-specific context before embedding and combines embeddings with BM25. Anthropic reports that this reduces failed retrievals by 49 percent, and by 67 percent when combined with reranking. Those are Anthropic's results on its own test sets. Your corpus may behave differently, so measure retrieval separately from answer quality.

A practical rule: build a small test set of real questions with the passages that should be retrieved, and check recall before you tune anything else.

When MCP is the answer

• The data lives in a system with its own permissions: a ticket tracker, CRM, calendar or database.
• The agent should act, not only read: file a ticket, update a record, run a query.
• Freshness matters. Fetching at call time beats an index that is a day old.
• You want one connection that works in several AI apps. The MCP site names assistants like Claude and ChatGPT and tools such as Visual Studio Code and Cursor as clients.

MCP has costs too

Tool definitions occupy context. Claude Code defers MCP tool definitions by default, so only names and server instructions are loaded until a tool is used, but its cost page still advises disabling unused servers and preferring CLI tools like gh and aws where they exist, since they add no per-tool listing.

Security needs care. The spec says tools represent arbitrary code execution and hosts must obtain explicit user consent before invoking them. Claude Code's docs warn that servers that fetch external content can expose you to prompt injection. For the full comparison with direct API calls, see MCP vs API.

Using both together

The most common pattern is a retrieval system behind an MCP server.

1. You build an index over your documents (the RAG part).
2. You wrap a search_docs function in an MCP server, with a clear description and a small result limit.
3. Any MCP client can now call it, in Claude Code, an IDE or a chat app.
4. The same server can offer other tools: get_ticket, create_ticket.

The agent decides when to search, can search again with a better query when the first results are weak, and can act on what it finds. That agentic loop often beats a fixed pipeline that retrieves once and answers once, because the model can see that a result is poor and try again. It also costs more calls. Set a budget on the number of tool calls per question.

You can also skip the server and run retrieval before the model call. That is simpler, cheaper per question and easier to evaluate, and it suits fixed-shape tasks such as "answer from the FAQ."

A worked example

A support team has 3,000 help articles, a ticketing system and a billing system.

• Answering "how do I reset my password?" The question maps to the help center. RAG over the 3,000 articles fits. No tools are needed.
• Answering "why was I charged twice?" This needs the customer's invoices. MCP to the billing system fits, with read-only tools.
• Refunding a charge. MCP with a write tool, behind explicit approval.
• "Why was I charged twice, and what does our policy say?" Both: a billing tool for the facts and retrieval for the policy text.

The pattern: documents point to RAG, live systems and actions point to MCP, and real questions often need both.

How to test each

The two need different tests, which is one more sign they are different things.

For RAG, test retrieval first. Write 30 real questions, note which passages should come back, and measure how often they appear in the top results. Then test the answers. If retrieval recall is low, no prompt change will rescue the answers.

For MCP, test tool choice and safety. Give the agent ten tasks and see which tools it calls, with what arguments, and whether it stays inside its permissions. Try a task designed to tempt it into a write action, and confirm that your approval prompt fires. Check the usage screen afterward to see what the tool definitions and results cost.

Keep both test sets. When a vendor updates a model or you change a server, rerun them.

Choosing

Choose RAG when:

• Your corpus is large, mostly text and changes slowly.
• You need cited passages.
• You can invest in an evaluation set for retrieval.

Choose MCP when:

• The data sits in live systems with access rules.
• The agent needs to act.
• You want a connection many clients can reuse.

Choose neither when:

• Your whole knowledge base fits in the prompt, cached.
• A script with a single API call does the job.

A short checklist

• Does the whole corpus fit in context? If so, start there.
• Do you need actions? Then you need tools, which means MCP or direct API calls.
• Do you have 30 or more real questions to test retrieval?
• Are exact identifiers common in your data? Add lexical search to embeddings.
• Is every MCP server one you trust, and is write access gated?
• Have you capped tool calls per question?

The skip-retrieval option, priced

Anthropic's advice to skip retrieval for a knowledge base under 200,000 tokens needs a price tag, since "put it all in the prompt" costs tokens on every call. Use Sonnet 5.5's list prices of $2 per million input tokens and $0.10 per million for a cache read.

At 10,000 questions a month, that is $4,000 uncached, $200 with a warm cache, and $100 with retrieval of 5,000 tokens a question, before you count the cost of building and running the index. Two things follow. Caching closes most of the gap, so the "skip retrieval" option is cheaper than it looks if your prefix is stable and your traffic keeps the cache warm. And retrieval still wins on cost at very high volume or for knowledge bases far larger than the window, which is why it has not gone away. The cache write price and its time limit matter for a quiet service, because a cache that expires between questions is paid for again each time.

The traffic is an example. Replace it with your own and recompute.

Index maintenance, the part people forget

A RAG pipeline is a small data system, and it needs the habits of one.

• Re-indexing. Decide how often documents change and when the index follows. A help article edited at noon and indexed at midnight gives stale answers for twelve hours.
• Deletes. When a document is removed, its chunks must go too, or the agent will quote a page that no longer exists.
• Permissions. If some documents are restricted, the index must respect that. A retriever that ignores access rules leaks text that a person could not open.
• Versions. Record which embedding model built the index. Changing the model means rebuilding every vector.
• Evaluation. Keep the 30 questions from the test section and rerun them after every change to chunking, the embedding model or the retriever.

An MCP server in front of a live system has fewer of these chores, because the source system holds the data and the permissions. That is part of why MCP suits live records and RAG suits documents that change slowly.

Questions people ask

Is RAG obsolete now that context windows reach a million tokens? No. The window decides when you can skip it, and cost and corpus size decide when you cannot.

Can an MCP server do retrieval? Yes. A search_docs tool in a server is the common way to put retrieval behind MCP, as the section above describes.

Which one cites sources? RAG can return the passage behind an answer by design. With MCP, citation depends on what the tool returns, so ask for sources in the tool's output.

Our take

From the documents, comparing RAG with MCP is like comparing a search algorithm with a USB port. One decides what to show the model, the other connects the model to things. Treat them as layers. Start with the smallest design that works: put documents in the prompt if they fit, add retrieval when they do not, and add MCP when the agent must reach live systems or act.

We did not build or benchmark either for this page. The retrieval figures are Anthropic's, from its own post, and the protocol facts are from the MCP documentation, both read on 31 October 2026.

Related

• MCP vs API
• Skills vs MCP servers
• MCP vs A2A
• MCP servers
• Best MCP servers
• How we compare