Spaces:
Running
on
Zero
Running
on
Zero
Commit
·
e3dcfdd
1
Parent(s):
10a9ffa
describe
Browse files
app.py
CHANGED
@@ -133,10 +133,18 @@ def generate_response(image):
|
|
133 |
return {}
|
134 |
|
135 |
|
136 |
-
title = "ColPali Query Generator"
|
137 |
-
description = """
|
|
|
|
|
|
|
|
|
|
|
138 |
|
139 |
This [blog post](https://danielvanstrien.xyz/posts/post-with-code/colpali/2024-09-23-generate_colpali_dataset.html) gives an overview of how you can use this kind of approach to generate a full dataset for fine-tuning ColPali models.
|
|
|
|
|
|
|
140 |
"""
|
141 |
|
142 |
demo = gr.Interface(
|
|
|
133 |
return {}
|
134 |
|
135 |
|
136 |
+
title = "ColPali fine-tuning Query Generator"
|
137 |
+
description = """[ColPali](https://huggingface.co/papers/2407.01449) is a very exciting new approach to multimodal document retrieval which aims to replace existing document retrievers which often rely on an OCR step with an end-to-end multimodal approach.
|
138 |
+
To train ColPali models, we need a dataset of image-text pairs which represent the document images and the relevant text queries which those documents should match.
|
139 |
+
To make the ColPali models work even better we might want a dataset of query/image document pairs related to our domain or task.
|
140 |
+
One way in which we might go about generating such a dataset is to use an VLM to generate synthetic queries for us.
|
141 |
+
This space uses the [Qwen/Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) to generate queries for a document, based on an input document image.
|
142 |
+
|
143 |
|
144 |
This [blog post](https://danielvanstrien.xyz/posts/post-with-code/colpali/2024-09-23-generate_colpali_dataset.html) gives an overview of how you can use this kind of approach to generate a full dataset for fine-tuning ColPali models.
|
145 |
+
|
146 |
+
If you want to convert a PDF(s) to a dataset of page images you can try out the [ PDFs to Page Images Converter](https://huggingface.co/spaces/Dataset-Creation-Tools/pdf-to-page-images-dataset) Space.
|
147 |
+
|
148 |
"""
|
149 |
|
150 |
demo = gr.Interface(
|