Tool Learning with Foundation Models: Frameworks, Datasets, and Open‑Source Toolkits

Background – Large foundation models have demonstrated strong semantic understanding, world knowledge, and reasoning abilities, enabling them to use external tools to solve complex tasks. The goal of tool learning is to let these models follow human instructions to operate tools, presenting both opportunities and challenges.

Categories – Tool learning can be divided into tool‑augmented learning (tools enhance the model) and tool‑oriented learning (the model optimizes tool usage). Typical scenarios include book recommendation, robot cooking, and invoking existing APIs for tasks like image generation.

Framework – A general tool‑learning framework mirrors an MDP and consists of four components: a tool set (physical, GUI, or programmatic tools), a controller (planning and invoking tools), a perceiver (collecting feedback), and an environment (physical or virtual). The interaction loop involves instruction sending, planning, feedback collection, summary generation, model update, and iteration.

Key Modules – The framework includes intent understanding (handling ambiguous queries), tool understanding (zero‑shot and few‑shot prompting), planning & reasoning (introspective vs. extrospective), and training strategies (behavior cloning and reinforcement learning).

WebCPM – An open‑source Chinese LFQA system that reproduces WebGPT. It provides a web‑search model, action prediction, query generation, fact extraction, and information synthesis modules, along with a high‑quality dataset of 5.5k QA pairs and over 100k real search actions.

BMTools – An extensible tool‑learning platform supporting custom Python tools, integration with ChatGPT plugins, local models (e.g., CPM‑Bee), planning frameworks (LangChain, BabyAGI, AutoGPT), and over 30 ready‑to‑use tools such as weather APIs, search engines, and image generators.

ToolBench – A benchmark and toolkit for building large‑scale instruction‑tuned data for tool usage. It offers single‑ and multi‑tool configurations, detailed reasoning traces, and supports real‑world tasks like weather retrieval and stock updates. The released dataset contains 31.2 k API calls and 9.8 k examples, enabling fine‑tuning of models such as ToolLLaMA.

Additional Resources – The article lists a curated collection of tool‑learning papers, links to the WebCPM and BMTools repositories, and provides Q&A from a recent sharing session.