Enhancing Language and Vision Models with External Knowledge and Tools: OREO‑LM, REVEAL, and AVIS

Background Modern neural models excel at memorizing factual knowledge but struggle with logical and discrete reasoning, prompting research into integrating external symbolic resources such as knowledge graphs, databases, and tool APIs.

OREO‑LM: Knowledge‑Graph Reasoning for Language Models OREO‑LM inserts knowledge‑graph interaction layers between frozen T5 blocks, allowing the model to issue entity queries, retrieve relation embeddings, and perform differentiable random walks on the graph. Special tokens (RET, T‑ENT) mediate the interaction, and multiple interaction layers enable multi‑hop reasoning. Experiments on multi‑hop QA show substantial performance gains and improved interpretability.

REVEAL: Multi‑Source Retrieval for Vision‑Language Pretraining REVEAL builds a unified memory that stores compressed embeddings of entities from diverse corpora (WIT, CC12M, Wikidata, VQA‑2). A perceived model (Transformer decoder) compresses high‑dimensional inputs into low‑dimensional key‑value pairs. During training, queries retrieve top‑k relevant knowledge items, whose attention scores are fused with the language input (Attentive Knowledge Fusion). This enables end‑to‑end pretraining and strong results on downstream VQA benchmarks, even when large portions of the knowledge graph are removed.

AVIS: Dynamic Tree‑Based Tool Calling for Large Models AVIS proposes a planning‑and‑execution framework where a planner predicts which API to invoke and formulates the corresponding query. A reasoner evaluates the result and feeds back to the planner, allowing backtracking and dynamic tool selection. The system integrates search, captioning, object detection, and external APIs (search engines, calculators) to answer complex multimodal questions without fine‑tuning the underlying language model.

Experimental Findings Across all three systems, incorporating external knowledge or tools yields notable improvements on multi‑hop QA, visual‑question answering, and knowledge‑intensive benchmarks. Ablation studies confirm that the gains stem from explicit reasoning over the external resources rather than mere parameter scaling.

Conclusion The work demonstrates three complementary approaches to bridge the gap between neural and symbolic AI: (1) differentiable graph reasoning within language models, (2) unified multimodal memory for scalable knowledge retrieval, and (3) a dynamic planning framework for tool‑augmented inference, collectively advancing the capability of smaller models to solve complex tasks.