Commit
·
a8a6bf2
1
Parent(s):
fae0273
docs: update vdr info
Browse filesSigned-off-by: jupyterjazz <[email protected]>
- README.md +4 -0
- vidore_eval.md +0 -26
README.md
CHANGED
@@ -308,6 +308,10 @@ code_embeddings = model.encode(
|
|
308 |
</details>
|
309 |
|
310 |
|
|
|
|
|
|
|
|
|
311 |
## License
|
312 |
|
313 |
This model is licensed to download and run under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/deed.en). It is available for commercial use via the [Jina Embeddings API](https://jina.ai/embeddings/), [AWS](https://longdogechallenge.com/), [Azure](https://longdogechallenge.com/), and [GCP](https://longdogechallenge.com/). To download for commercial use, please [contact us](https://jina.ai/contact-sales).
|
|
|
308 |
</details>
|
309 |
|
310 |
|
311 |
+
## Jina-VDR
|
312 |
+
We’re releasing Jina VDR, a multilingual, multi-domain benchmark for visual document retrieval, alongside jina-embeddings-v4. The task collection can be viewed [here](https://huggingface.co/collections/jinaai/jinavdr-visual-document-retrieval-684831c022c53b21c313b449), and evaluation instructions can be found [here](https://github.com/jina-ai/jina-vdr).
|
313 |
+
|
314 |
+
|
315 |
## License
|
316 |
|
317 |
This model is licensed to download and run under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/deed.en). It is available for commercial use via the [Jina Embeddings API](https://jina.ai/embeddings/), [AWS](https://longdogechallenge.com/), [Azure](https://longdogechallenge.com/), and [GCP](https://longdogechallenge.com/). To download for commercial use, please [contact us](https://jina.ai/contact-sales).
|
vidore_eval.md
DELETED
@@ -1,26 +0,0 @@
|
|
1 |
-
# How to run the Vidore Evaluation
|
2 |
-
|
3 |
-
If you want to run the vidore evaluation on the jina-embeddings-v4 model (and on the Document Retrieval Benchmark curated by Jina AI), you need to install requirements in [this fork/branch](https://github.com/jina-ai/vidore-benchmark-fork/tree/feat-add-jina-embeddings) (these changes should be merged with the source code of Vidore soon).
|
4 |
-
|
5 |
-
```
|
6 |
-
pip install vidore-benchmark[jina-v4]
|
7 |
-
```
|
8 |
-
|
9 |
-
You can run the evaluation with the following command:
|
10 |
-
|
11 |
-
```
|
12 |
-
vidore-benchmark evaluate-retriever \
|
13 |
-
--model-class jev4 \
|
14 |
-
--model-name jinaai/jina-embeddings-v4 \
|
15 |
-
--collection-name jinaai/jinavdr-visual-document-retrieval-684831c022c53b21c313b449 \
|
16 |
-
--dataset-format qa \
|
17 |
-
--split test
|
18 |
-
```
|
19 |
-
|
20 |
-
## Evaluate Pure Text Retrieval Models on Refined Vidore Tasks
|
21 |
-
|
22 |
-
The original Vidore dataset contain multiple text chunks per image to evaluate text retrieval models on them.
|
23 |
-
Those text chunks are extracted from the document pages using different tools like [Unstructured](https://github.com/Unstructured-IO/unstructured), OCR models, and LLMs.
|
24 |
-
For evaluating text retrieval models on our filtered versions of the Vidore datasets, you can use the datasets in the collection `https://huggingface.co/collections/jinaai/jina-vdr-vidoreocr-tasks-6852cfc55ccf837e7fecfa1b`.
|
25 |
-
|
26 |
-
It is also possible to evaluate jina-embeddings-v4 and other vision retrieval models on them. This however takes more time and should lead to the same evaluation results as running the vesions of the datasets in the Jina VDR collection.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|