RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval‑Augmented Generation

When building applications based on large language models (LLMs), developers often face the problem of how to let the LLM intelligently and accurately use external tools or invoke functions. As the tool library grows, the LLM can suffer from "prompt bloat" and "selection difficulty" because the prompt becomes excessively long and the model may choose the wrong tool.

The paper RAG‑MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval‑Augmented Generation proposes a framework called RAG‑MCP that integrates Retrieval‑Augmented Generation (RAG) with the Model Context Protocol (MCP) to address this issue.

RAG‑MCP workflow:

Tool knowledge base creation : Store all tool descriptions (e.g., MCP function signatures, usage examples) in an external, searchable memory bank and index them semantically (e.g., vector embeddings).

Query‑driven tool retrieval : When a user query arrives, a lightweight retriever (small encoder + vector search) selects the top‑K most relevant tool candidates instead of sending the whole tool set to the LLM.

Lightweight LLM inference : Only the selected K tool descriptions are injected into the LLM prompt (or provided via Function Calling API), dramatically reducing the amount of information the LLM must process.

(Figure 1 shows the comparison between traditional MCP and RAG‑MCP, highlighting the additional retrieval step that optimizes the LLM input.)

(Figure 2 illustrates the three‑step RAG‑MCP process: query encoding → vector search & candidate MCP verification → LLM execution with selected MCP.)

The authors evaluated RAG‑MCP with a series of experiments, including an "MCP stress test" that varied the number of tools from 1 to 11,100. Results show that RAG‑MCP achieves a tool‑selection accuracy of 43.13%, far surpassing the baseline methods (Actual Match 18.20% and Blank Conditioning 13.62%). It also reduces average prompt tokens from 2,133.84 (Blank Conditioning) to 1,084.00, cutting token consumption by roughly half.

(图3的热力图直观显示了，随着工具池的增大，选择成功率会下降，但RAG-MCP能够在一定程度上缓解这种趋势，尤其是在中小型工具池中表现优异。黄色代表成功，紫色代表失败。)

From an architect’s perspective, RAG‑MCP brings several benefits:

Scalability : Decouples tool discovery from execution, allowing the system to handle far more tools than the LLM’s context window.

Efficiency & Cost : Fewer prompt tokens mean lower API costs and faster responses.

Maintainability & Flexibility : Updating or adding tools only requires changes to the external index, not retraining the LLM.

Modularity : Retrieval and LLM modules can be optimized independently.

Robustness : Dynamic retrieval ensures each dialogue turn receives the most relevant tool context.

The paper also discusses limitations: when the tool pool reaches extreme sizes, retrieval precision and latency become new bottlenecks, suggesting future work on hierarchical indexing and adaptive retrieval strategies, as well as handling multi‑tool collaborative tasks.

Overall, RAG‑MCP offers a practical, research‑backed approach to enable LLM agents to remain focused and efficient in increasingly large tool ecosystems, paving the way for more capable AI assistants.