1 7 12

s k

madstuntman11

AI & ML interests

None yet

Recent Activity

liked a model 7 days ago

lightonai/Reason-ModernColBERT

liked a model 27 days ago

MrLight/dse-qwen2-2b-mrl-v1

upvoted an article 27 days ago

Visual Document Retrieval Goes Multilingual

View all activity

Organizations

None yet

madstuntman11's activity

liked a model 7 days ago

lightonai/Reason-ModernColBERT

liked a model 27 days ago

MrLight/dse-qwen2-2b-mrl-v1

Visual Document Retrieval • Updated Feb 26 • 2.29k • 59

upvoted an article 27 days ago

Article

Visual Document Retrieval Goes Multilingual

and 1 other •

Jan 10

• 74

liked a model 28 days ago

reducto/RolmOCR

Image-Text-to-Text • Updated Apr 2 • 134k • 411

upvoted an article 2 months ago

Article

DeepSearch Using Visual RAG in Agentic Frameworks 🔎

and 1 other •

Mar 21

• 33

reacted to merve's post with ❤️ 7 months ago

Post

3893

If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗

Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲

How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝

This is much faster + you do not lose out on any information + much easier to maintain too! 🥳

Multimodal RAG merve/multimodal-rag-66d97602e781122aae0a5139 💬
Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e 📖

2 replies

reacted to merve's post with ❤️ 7 months ago

Post

5662

I have put together a notebook on Multimodal RAG, where we do not process the documents with hefty pipelines but natively use:
- vidore/colpali for retrieval 📖 it doesn't need indexing with image-text pairs but just images!
- Qwen/Qwen2-VL-2B-Instruct for generation 💬 directly feed images as is to a vision language model with no processing to text!
I used ColPali implementation of the new 🐭 Byaldi library by @bclavie 🤗
https://github.com/answerdotai/byaldi
Link to notebook: https://github.com/merveenoyan/smol-vision/blob/main/ColPali_%2B_Qwen2_VL.ipynb