VisualRAG 🖼️🔍

A lightweight, modular Multimodal Retrieval-Augmented Generation (RAG) pipeline for image-and-text understanding.

VisualRAG bridges the gap between large vision-language models (VLMs) and real-world document retrieval. Instead of relying solely on text, it indexes image content using CLIP/SigLIP embeddings, retrieves relevant visual context, and feeds it to a generative model to produce grounded, factual answers.

Motivation: Existing RAG systems are text-only. But a lot of knowledge lives in figures, charts, diagrams, and scanned PDFs. VisualRAG treats images as first-class citizens.

✨ Key Features

• Flexible image encoders: CLIP ViT-L/14, SigLIP, OpenCLIP — swappable via config
• Efficient vector store: FAISS-based ANN retrieval with optional GPU acceleration
• Multi-source document parsing: PDF pages, image folders, HuggingFace datasets
• Pluggable generators: OpenAI GPT-4o, local LLaVA-1.6, InternVL2
• Evaluation suite: CLIPScore, BLEU, ROUGE, and custom VQA accuracy metrics
• Gradio demo: One-command interactive web interface

🏗️ Architecture

Query (text/image)
       │
       ▼
  ┌─────────────┐     ┌──────────────────┐
  │ Query Encoder│────▶│  FAISS Index     │
  │ (CLIP/SigLIP)│     │  (image embeds)  │
  └─────────────┘     └────────┬─────────┘
                               │  top-k images
                               ▼
                      ┌──────────────────┐
                      │   VLM Generator  │
                      │ (GPT-4o/LLaVA)   │
                      └────────┬─────────┘
                               │
                               ▼
                           Answer

🚀 Quick Start

Installation

git clone https://github.com/suncatchin/visual-rag
cd visual-rag
pip install -e ".[full]"

Build an index from a folder of images

from visual_rag import VisualRAGPipeline

pipeline = VisualRAGPipeline(
    encoder="clip-vit-large-patch14",
    generator="gpt-4o",
)

# Index a directory of images
pipeline.index_directory("./my_images/")

# Query
answer = pipeline.query("What does the training loss curve show?")
print(answer)

Index a PDF document

pipeline.index_pdf("report_2024.pdf", extract_images=True)
answer = pipeline.query("What is the main conclusion of the ablation study?")

Run the Gradio demo

python examples/gradio_demo.py --index ./my_images/

📂 Project Structure

visual_rag/
├── encoder.py          # Image/text encoders (CLIP, SigLIP, OpenCLIP)
├── retriever.py        # FAISS vector store and retrieval
├── generator.py        # VLM-based answer generation
├── pipeline.py         # End-to-end RAG pipeline
├── document_parser.py  # PDF and image folder parsing
├── metrics.py          # Evaluation metrics
└── utils.py            # Helpers and config
examples/
├── demo.py             # Script-based demo
└── gradio_demo.py      # Interactive web demo
tests/
├── test_encoder.py
└── test_retriever.py

📊 Benchmark

Evaluated on InfoVQA and DocVQA benchmarks:

Method	InfoVQA Acc.	DocVQA ANLS
Text-only RAG	41.2	0.523
VisualRAG (CLIP-L)	58.7	0.681
VisualRAG (SigLIP)	61.3	0.704
VisualRAG + reranking	63.9	0.718

⚙️ Configuration

# config.yaml
encoder:
  model: "clip-vit-large-patch14"
  device: "cuda"
  batch_size: 64
retriever:
  top_k: 5
  index_type: "IVFFlat"   # or "Flat", "HNSW"
  nprobe: 32
generator:
  model: "gpt-4o"
  max_tokens: 512
  temperature: 0.1

📖 Citation

If you use VisualRAG in your research, please cite:

@misc{xu2024visualrag,
  title={VisualRAG: Multimodal Retrieval-Augmented Generation with Visual Context},
  author={Xu, Haowen},
  year={2024},
  url={https://github.com/suncatchin/visual-rag}
}

🤝 Contributing

Contributions are welcome! Please open an issue or submit a PR. See CONTRIBUTING.md for guidelines.

📄 License

MIT License — see LICENSE for details.