Enhancing Vision and Language Models with External Knowledge Graphs and Tool Integration

The presentation introduces the motivation for enhancing visual and language models with external knowledge and tools, highlighting the limitations of pure neural models in logical and discrete reasoning tasks.

Three research works are described:

OREO‑LM : integrates knowledge‑graph reasoning into language models by adding interaction layers that allow the model to perform graph walks and retrieve relational information in a differentiable manner.

REVEAL : extends visual‑language models with a unified multi‑source memory that stores embeddings from various knowledge bases, enabling end‑to‑end training of retrieval and attention‑fusion mechanisms for multimodal question answering.

AVIS : proposes a dynamic tree‑decision framework that lets large language models plan and execute API calls to external tools (search engines, calculators, etc.) without fine‑tuning, supporting iterative reasoning and tool selection.

Experimental results show that OREO‑LM improves multi‑hop QA performance on T5, REVEAL achieves strong results on multimodal datasets such as WIT, CC12M, Wikidata, and VQA‑2, and AVIS approaches 50% accuracy on challenging infoseek benchmarks by combining tool use with planning.

The article concludes that combining symbolic knowledge sources with neural models, using differentiable interaction layers and dynamic tool‑calling strategies, can significantly boost reasoning capabilities and robustness of AI systems while remaining trainable on unlabeled data.