Multimodal RAG: Implementation Paths and Development Prospects

This presentation focuses on the implementation paths and development prospects of Multimodal RAG (Retrieval Augmented Generation). The core topics cover five areas: semantic extraction-based multimodal RAG, VLM-based multimodal RAG, how to scale VLM-based multimodal RAG, technology roadmap selection, and Q&A session.

Three Main Technical Approaches:

1. Traditional Object Recognition (OCR-based approach) : Uses image recognition technologies like OCR to extract text, tables, and images from documents, then converts these objects into text formats for retrieval and analysis. This method involves document structure recognition, text transcription via OCR, and specialized model parsing for charts and tables.

2. Transformer Architecture Approach : Uses encoder-decoder architecture to encode entire documents and transform encoded information into readable text. This method better captures contextual dependencies and improves information coherence.

3. Visual Language Model (VLM) Approach : Directly uses VLM to process multimodal data, converting documents, images, or videos into vectors (Patch Embedding) for building finer document embeddings. Using multi-vectors (tensors) is preferred over single vectors to reduce information loss.

ColPali Method: The presentation discusses ColPali, which uses context-based late interaction. It converts multimodal documents into multi-dimensional vectors and uses similarity matching for answer generation. A PDF document is split into 1024 patches, each represented by a 128-dimensional vector, forming a tensor with 1024 vectors.

Scaling VLM-based Multimodal RAG: The main challenge is increased data scale and tensor complexity. Solutions include tensor binarization and using Tensor Reranker for re-ranking. The Infinity database supports structured data, dense vectors, sparse vectors, tensors, and full-text search with fusion search capabilities.

Technology Selection: The two approaches (OCR-based and VLM-based) will coexist long-term. VLM is more suitable for documents with many abstract images, while traditional methods work better for other cases. Tensor-based late interaction will become the standard for future multimodal RAG.