corrected typos

app/src/content/article.mdx

@@ -44,9 +44,9 @@ import visualPoster from "./assets/images/visual-vocabulary-poster.png";
 import Accordion from '../components/Accordion.astro'
 
 <Sidenote>
-  Today, we release **FineVision**, a new multimodal dataset with **24 million samples**. We created FineVision by processing over 200 datasets containing 17M images, 90M question-answer turns, and 10B answer tokens, totaling **5TB of high-quality data**. Additionally, we extensively processed all datasets to unify their format, clean them of duplicates and poor data, and rated all 90M turns using 32B VLMs across 4 qualitative metrics with a score from 1-5 to enable the construction and study of individual training mixtures.
+  Today, we release **FineVision**, a new multimodal dataset with **24 million samples**. We created FineVision by collecting over 200 datasets containing 17M images, 89M question-answer turns, and 10B answer tokens, totaling **5TB of high-quality data**. Additionally, we extensively processed all datasets to unify their format, clean them of duplicates and poor data, and rated all turns using 32B VLMs across 4 qualitative metrics with a score from 1-5 to enable the construction and study of individual training mixtures.
   
-  To enable everyone to construct SOTA open VLMs, we ran extensive ablations on FineVision, and compared it to publicly available alternatives. Models trained on FineVision lead in performance over 11 common benchmarks compared against every baseline, thanks to FineVision’s scale and diversity of data.
+  To enable everyone to construct state-of-the-art open Vision-Language Models (VLMs), we ran extensive ablations on FineVision, and compared it to publicly available alternatives. Models trained on FineVision lead in performance over 11 common benchmarks compared against every baseline, thanks to FineVision’s scale and diversity of data.
 
   To use the dataset, simply load it with:
 
@@ -64,15 +64,16 @@ import Accordion from '../components/Accordion.astro'
 </Sidenote>
 
 ## Why this dataset?
-Even though open-weights Vision-Language Models (VLMs) are becoming ever more powerful, the accessibility of the training data used for these models is lagging behind. This data is often proprietary and inaccessible for the broader community. Projects like The Cauldron, LLaVa and Cambrian aim to provide such datasets, but get quickly outpaced by the speed of the field and the emergence of novel applications for VLMs, like agentic tasks.
+Even though open-weights Vision-Language Models are becoming ever more powerful, the accessibility of the training data used for these models is lagging behind. This data is often proprietary and inaccessible for the broader community. Projects like The Cauldron, LLaVa and Cambrian aim to provide such datasets, but get quickly outpaced by the speed of the field and the emergence of novel applications for VLMs, like agentic tasks.
 For FineVision we set out to combine and unify existing available data sources to create a large and high-quality dataset. As a first step we need to collect and standardize the datasets.
 
 ## How did we build FineVision?
-FineVision was a giant act of data curation. We started by collecting publicly available datasets, and augmenting underrepresented categories. We then evaluated all datasets for duplicated data internally and benchmarks contamination. This data is then cleaned and rated, before being added to the final mixture.
+FineVision was a giant act of data curation. We started by collecting publicly available datasets, and augmenting underrepresented categories. We then evaluated all datasets for duplicated data internally and benchmark contamination. This data is then cleaned and rated, before being added to the final mixture.
 
 ### Data Collection
-We manually collected over **200 image-text datasets** from various publicly available sources and processed them to unify their formatting. On top of that, some datasets are not presented in chat form, so we converted them into question-answer pairs. In some cases, this goes as far as creating questions for all samples synthetically. Finally, we created new subsets for underrepresented domains, such as GUI-oriented data. To address this gap, we added a new dataset which is compiled from existing GUI datasets, after applying chat normalization and unifying the action space to convert their specific formats into a more general GUI action space.
+We manually collected over **200 image-text datasets** from various publicly available sources and processed them to unify their formatting. On top of that, some datasets are not presented in chat form, so we converted them into question-answer pairs. In some cases, this goes as far as synthetically creating questions for all samples. Finally, we adressed underrepresented domains, such as GUI-oriented data. To fill this gap, we create and add a new dataset which was compiled from existing GUI datasets, after applying chat normalization and unifying the action space to convert their specific formats into a more general GUI action space.
 
+---
 <Wide>
 <Accordion size="big" title="FineVision Subsets - click to see more">
 |Subset Name                           |Total Images|Total Samples|Total Turns|Total Question Tokens|Total Answer Tokens|Category              |Source                                                             |
@@ -262,24 +263,25 @@ We manually collected over **200 image-text datasets** from various publicly ava
 |text_theoremqa                        |0           |800          |800        |50,065               |3,468              |Text-only             |[@chen_theoremqa_2023]                                             |
 |text_wizardlm_evol                    |0           |69,999       |69,999     |7,753,963            |21,955,856         |Text-only             |[@noauthor_wizardlmteamwizardlm_evol_instruct_70k_2024]            |
 |text_OpenMathInstruct-2               |0           |1,000,000    |1,000,000  |74,905,850           |413,132,418        |Text-only             |[@toshniwal_openmathinstruct-2_2024]                               |
-|**Totals**                            |17,372,293  |24,322,193   |88,928,343 |3,168,958,417        |9,459,677,828      |                      |                                                                   | 
+|**Totals**                            |**17,372,293**  |**24,322,193**   |**88,928,343** |**3,168,958,417**        |**9,459,677,828**      |                      |                                                                   | 
 
 </Accordion>
 </Wide>
+---
 
 ### Cleaning
 After gathering all the sub-datasets, every turn is cleaned. We removed all individual turns whose combined question and answer length exceeds 8192 tokens. We resize big images to have a longest side of 2048 pixels while keeping the aspect ratio, and discard samples with corrupted images.
 
 ### Rating
 Finally, we rate every single turn in our dataset across 4 axes.
-For this, we used a LLM and VLM-as-a-judge pipeline (using Qwen3-32B and Qwen2.5VL-32B), to rate every turn on a scale from 1-5 in these 4 categories:
-- Text Formatting Quality: How is the quality of the answer both linguistically and structurally? (Question and Answer)
-- Question-Answer Relevance: Does the answer properly respond to the question? (Question and Answer)
-- Visual Dependency: How much does the question depend on visual information to be answered? (Question only)
-- Image-Question Correspondence: How well does the image support answering the question? (Image and Question)
-
-This is the distribution of scores across the different filters for FineVision.
-| Filter               	| 1   	| 2   	| 3   	| 4    	| 5    	|
+For this, we used a LLM and VLM-as-a-judge pipeline (using [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) and [Qwen2.5VL-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct)), to rate every turn on a scale from 1-5 in these 4 categories:
+- **Text Formatting Quality**: How is the quality of the answer both linguistically and structurally? (Question and Answer)
+- **Question-Answer Relevance**: Does the answer properly respond to the question? (Question and Answer)
+- **Visual Dependency**: How much does the question depend on visual information to be answered? (Question only)
+- **Image-Question Correspondence**: How well does the image support answering the question? (Image and Question)
+
+This is the distribution of scores across the different filters for FineVision:
+| **Filter**               	| **1**   	| **2**   	| **3**   	| **4**    	| **5**    	|
 |-----------------------|-----	|-----	|-----	|-----	|-----	|
 | Formatting        		| 0.5	  | 0.7 	| 1.1 	| 77.5 	| 20.3 	|
 | Relevance         		| 2.9 	| 0.5 	| 14.7	| 16.5 	| 65.4 	|
@@ -287,15 +289,17 @@ This is the distribution of scores across the different filters for FineVision.
 | Image Correspondence 	| 8.1 	| 3.6 	| 17.3	| 26.8 	| 44.1 	|
 
 ### FineVision Base Dataset
-We classify FineVision’s subsets into 9 categories: Captioning & Knowledge, Chart & Table, General VQA, Grounding & Counting, Mathematics, Naive OCR, OCR QA, Science, Text-only.
+We classify FineVision’s subsets into 9 categories: Captioning & Knowledge, Chart & Table, General VQA, Grounding & Counting, Mathematics, Naive OCR, OCR QA, Science and Text-only <a href="#pie">(Fig. 1)</a>.
 
-There are multiple ways to count the data in a multimodal dataset. The most common are the number of samples and the number of images. Additionally, a single sample can consist of multiple question/answer pairs in the form of a multi-turn conversation. Similarly to text-only datasets, the number of answer tokens is also interesting, since these are the tokens the model is actually trained on. We count all these characteristics for FineVision and arrive at 17.3M images, 24.3M samples, 88.9M turns, and 9.5B answer tokens. Based on these 4 distributions, multiple different mixtures are possible. In conjunction with the provided ratings, we encourage the community to create their own mixtures and experiment with the data. For example, large categories could be downsamples, while high-quality data could be upsampled with techniques such as rephrasing.
+There are multiple ways to count the data in a multimodal dataset. The most common are the number of samples and the number of images. Additionally, a single sample can consist of multiple question/answer pairs in the form of a multi-turn conversation. Similarly to text-only datasets, the number of answer tokens is also interesting, since these are the tokens the model is actually trained on. We count all these characteristics for FineVision and arrive at **17.3M images**, **24.3M samples**, **88.9M turns**, and **9.5B answer tokens**. Based on these 4 distributions, multiple different mixtures are possible. In conjunction with the provided ratings, we encourage the community to create their own mixtures and experiment with the data. For example, large categories could be downsamples, while high-quality data could be upsampled.
 After collecting and processing the data, we run multiple experiments and ablations to provide practical recommendations on how to train small, data-centric VLMs.
 
+---
 <br/>
 <Wide>
-  <HtmlEmbed src="d3-pie.html" desc="Figure 1: Distribution of Categories in FineVision by Answer Tokens, Number of Samples, Turns, and Images. While the distributions differ a bit with the different metrics, FineVision provides a good baseline mixture especially when judging by the number of images in the individual categories. Samples from Chart & Table usually lend themselves well to multi turn conversations, since multiple similar questions can be asked for a single Chart. Samples from OCR QA often have a lot of answer tokens, since they aim at detailed document understanding, which are rarely answered with a short sentence." align="center" />
+  <HtmlEmbed src="d3-pie.html" id="pie" desc="Figure 1: Distribution of Categories in FineVision by Answer Tokens, Number of Samples, Turns, and Images. While the distributions differ a bit with the different metrics, FineVision provides a good baseline mixture especially when judging by the number of images in the individual categories. Samples from Chart & Table usually lend themselves well to multi turn conversations, since multiple similar questions can be asked for a single Chart. Samples from OCR QA often have a lot of answer tokens, since they aim at detailed document understanding, which are rarely answered with a short sentence." align="center" />
 </Wide>
+---
 
 ## Experimental Setup
 To ensure a fair comparison between different configurations, we use the same setup and evaluations for all of our ablations. This enables us to compare FineVision to other publicly available datasets as well as experiment with different intra-dataset configurations.
@@ -306,18 +310,18 @@ For all ablations and experiments, we train a 460M parameter VLM, since it provi
 ### Baseline Datasets
 We use 3 similar open source alternatives as baselines to compare our dataset to: **[The Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron)**, **[LLaVA-OneVision](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data)** and **[Cambrian-7M](https://huggingface.co/datasets/nyu-visionx/Cambrian-10M)**.
 
-| Name          | Samples	| Images  | Answer Tokens  | Turns |	
-|---------------|---------|---------|----------------|-------|
-| Cauldron      | 1.8M		| 2.0M    | 0.3B           | 27.8M |
-| Llava-Vision  | 3.9M		| 2.5M    | 1.0B           | 9.1M  |
-| Cambrian-7M   | 7M		  | 5.4M	  | 0.8B           | 12.2M |
-| FineVision		| 24.3M	  | 17.3M   | 9.5B	         | 88.9M |
+| Name          | Images  | Samples	| Turns |	Answer Tokens  
+|---------------|---------|---------|-------|----------------|
+| Cauldron      | 2.0M    | 1.8M		| 27.8M | 0.3B           | 
+| LLaVa-Vision  | 2.5M    | 3.9M		| 9.1M  | 1.0B           |
+| Cambrian-7M   | 5.4M	  | 7M		  | 12.2M | 0.8B           |
+| FineVision		| 17.3M   | 24.3M	  | 88.9M | 9.5B	         |
 
 ### Evaluations
-We utilize [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) during training to evaluate our ablations in a reproducible manner. We evaluate on a diverse set of 11 benchmarks: AI2D, ChartQA, DocVQA, InfoVQA, MME, MMMU, MMStar, OCRBench, ScienceQA, TextVQA and Seedbench. Since these benchmarks cover different topics and produce results on different scales, e.g. AI2D returns the accuracy of the exact matches (0-100), but MME returns a continuous score (0-2800), aggregating them is not trivial. In our ablations the relative performance between the different configurations matters, so we determine the rank of each model compared to the others in every benchmark at every training step and average it over all the benchmarks. This way we can judge where different configurations rank among each other over the course of training, and how big the difference between them is.
+We utilize [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) during training to evaluate our ablations in a reproducible manner. We evaluate on a diverse set of 11 benchmarks: AI2D, ChartQA, DocVQA, InfoVQA, MME, MMMU, MMStar, OCRBench, ScienceQA, TextVQA and Seedbench. Since these benchmarks cover different topics and produce results on different scales, e.g. AI2D returns the accuracy of the exact matches (0-1), but MME returns a continuous score (0-2800), aggregating them is not trivial. In our ablations the relative performance between the different configurations matters, so to provide a robuts summary metric we determine the rank of each model compared to the others in every benchmark at every training step and average it over all the benchmarks. This way we can judge where different configurations rank among each other over the course of training. To keep a sense of how big the absolute difference between models is, we also provide an average over all metrics and incorporate MME by normalizing it between 0 and 1.
 
 ### Training Configuration
-Each of our ablations trains said 460M model with a maximal image size of 1536x1536 pixel (without resizing smaller images) and a maximal input token length of 4096. This results in a maximum batch size of 2 for a single H100, which we adapt with 8 steps of gradient accumulation for an effective batch size of 16. In all single stage configurations we train for 20k Steps on 32 H100s for approximately 20h while evaluating all 11 benchmarks every 1k Steps. If not specified otherwise, the “Baseline” in our intra dataset ablations refers to a training run on the full unfiltered and unchanged dataset. In this configuration, a full epoch of the unfiltered FineVision dataset takes 12k steps.
+Each of our ablations trains said 460M model with a maximal image size of 1536x1536 pixel (without resizing smaller images) and a maximal input token length of 4096. This results in a maximum batch size of 2 for a single H100, which we adapt with 8 steps of gradient accumulation on each of the 32 GPUs for an effective batch size of 512. In all single stage configurations we train for 20k Steps on 32 H100s for approximately 20h while evaluating all 11 benchmarks every 1k Steps. If not specified otherwise, the “Baseline” in our intra dataset ablations refers to a training run on the full unfiltered and unchanged dataset. In this configuration, a full epoch of the unfiltered FineVision dataset takes 12k steps.
 
 ## Experiments
 While there are a lot of interesting questions that could be investigated, we mainly focus on the aspects of the training that are influenced by the data. Before we dive into the internal details of FineVision, let’s have a look at our performance against the baselines.
@@ -326,12 +330,12 @@ While there are a lot of interesting questions that could be investigated, we ma
 Here we see the first interesting trend: VLMs still benefit from training on a larger, more diverse dataset than what was available until today. FineVision doesn't lead the race in the first few thousand training steps, after all, it does include new tasks such as pointing and agentic browsing, so it shouldn't be better at first. But after seeing enough varied data, FineVision clearly shows the best performance across a wide set of benchmarks, which can be seen in its average ranking <a href="#against-baselines">(Fig. 2)</a>. One epoch of FineVision in our setup takes 12k training steps, so we train for close to 2 epochs in these ablations. Looking at the average benchmark, we can see how the models saturate around different points: 18k steps for cambrian, 12k for LLaVa and 7k for the cauldron.
 In particular, over 11 different benchmarks, FineVision achieves an average improvement of 40.7% over the Cauldron, 12.1% over Cambrian, and 46.3% over LLaVa, which increases to 51.3%, 18.6% and 58.0% when comparing the deduplicated versions of the datasets. Additionally, FineVision includes data for tasks such as agentic browsing, and counting and pointing, which are not part of the other baselines.
 
-<HtmlEmbed id="against-baselines" src="against-baselines.html" desc="Figure 2: Average Rank of Models trained on different open source datasets. FineVision shows both the highest average rank as well as the highes average over benchmarks." />
+<HtmlEmbed id="against-baselines" src="against-baselines.html" desc="Figure 2: Average Rank of Models trained on different open source datasets. FineVision shows both the highest average rank as well as the highest average over benchmarks." />
 
 ### How much test data is in publicly available datasets?
-We investigate data leakage by finding images from test sets that appear in the dataset. For this, we constructed an image deduplication pipeline. We used this pipeline to compare all images in FineVision to all images from 66 benchmarks from the lmms-eval framework.
+We investigate data leakage by finding images from test sets that appear in the dataset. For this, we constructed an image deduplication pipeline. We used this pipeline to compare all images in FineVision to all images of 66 image-text benchmarks from the lmms-eval framework.
 
-For the comparison, we embed the images using the SSCD descriptor, and compute cosine similarity between a given image in FineVision and all images from the test-set embeddings. Whenever a sample has a similarity higher than a threshold of **0.95** it is assumed to be a duplicate.
+For the comparison, we embed the images using the [SSCD descriptor](https://github.com/facebookresearch/sscd-copy-detection), and compute the cosine similarity between a given image in FineVision and all images from the test-set embeddings. Whenever a sample has a similarity higher than a threshold of **0.95** it is assumed to be a duplicate.
 
 While our tests with various thresholds show that this is still flagging more false-positives than false-negatives, given the scale of data we have, we preferred to err on the side of caution.
 
@@ -339,10 +343,15 @@ Below is an example of a correctly identified Duplicate (“Photo”), a false-p
 
 We open-source the deduplication pipeline [here](https://github.com/huggingface/large-scale-image-deduplication) as well as the precomputed test-set embedding’s [here](https://huggingface.co/datasets/HuggingFaceM4/lmms-eval-embeddings).
 
+---
 <br/>
 <Wide>
 <HtmlEmbed id="comparison" src="comparison.html" align="center" desc="Figure 3: Examples of the Deduplication Results."/>
 </Wide>
+---
+
+We repeated this deduplication procedure on all the baselines to analyse how contaminated they are. We found that all baselines contain between 2-3% images from test benchmarks, and removing them results in a performance drop of 2.4-2.8%. Interestingly, we find that for some benchmarks the difference is negligible, while other benchmarks suffer significantly. For example, after deduplicating, ScienceQA falls by 14.49% on average while OCRBench only drops by 1.08%.
+This deduplications also shows that FineVision contains the smallest relative amount of duplicated data at 1%, and also suffers the smallest performance drop over all benchmarks after deduplication at just 1.45%.
 
 | Name          | Samples	| Contamination Rate | Performance Drop |
 |---------------|---------|--------------------|------------------|
@@ -351,15 +360,14 @@ We open-source the deduplication pipeline [here](https://github.com/huggingface/
 | Cambrian-7M  	| 7.0M	  | 2.29%		           | 2.78%            |
 | FineVision		| 24.3M 	| 1.02%	      	     | 1.45%            |
 
-We repeated this deduplication procedure on all the baselines to analyse how contaminated they were. We found out that all baselines contain between 2-3% images from test benchmarks, and removing them results in a performance drop of 2.4-2.8%. Interestingly, we find that for some benchmarks the difference is negligible, while other benchmarks suffer significantly. For example, after deduplicating, ScienceQA falls by 14.49% on average while OCRBench only drops by 1.08%.
-This deduplications also shows that FineVision contains the smallest relative amount of duplicated data at 1%, and also suffers the smallest performance drop over all benchmarks after deduplication at just 1.45%.
-
 Additionally, we experimented with removing all found samples from all datasets to see if the outcome is different from <a href="#against-baselines">Fig. 2</a>, but we observe the same distribution <a href="#against-baselines-deduplicated">(Fig. 4)</a>.
 
+---
 <HtmlEmbed id="against-baselines-deduplicated" src="against-baselines-deduplicated.html" desc="Figure 4: Average Rank of Models trained on different deduplicated open source datasets. Even after deduplicating all dataset, FineVision shows the best performance." />
+---
 
 ### How diverse are the datasets?
-Similarly to the comparison of the size, we also wanted to evaluate the datasets for diversity. Evaluating the diversity of a dataset is a field of study for itself, which we will not dive into here, rather we borrow techniques from computer vision and use the already computed SSCD embeddings as a proxy of visual diversity. To not rely on a subsample of the dataset in estimating the diversity, we analyse the covariance metric of the full embeddings. From this covariance matrix, we can calculate the eigenvalues for analysis. We get the effective rank of the covariance matrix, which measures how uniformly the variance is distributed across dimensions, as well as the participation ratio, which measures how many dimensions actively contribute to the overall variance. To obtain a single **diversity score** for the datasets, we normalize the effective rank and participation ratio with the embedding dimension and compute their geometric mean. We observe that FineVision is not only the biggest, but also the most diverse dataset. Additionally, you can also clearly see that more images do not necessarily result in more diversity, since LLaVa is substantially less diverse than the Cauldron, even with more images.
+Similarly to the size comparison, we also wanted to evaluate the datasets for diversity. Evaluating the diversity of a dataset is a field of study for itself, which we will not dive into here, rather we borrow techniques from computer vision and use the already computed SSCD embeddings as a proxy of visual diversity. To not rely on a subsample of the dataset in estimating the diversity, we analyse the covariance metric of the full embeddings. From this covariance matrix, we can calculate the eigenvalues for analysis. We get the effective rank of the covariance matrix, which measures how uniformly the variance is distributed across dimensions, as well as the participation ratio, which measures how many dimensions actively contribute to the overall variance. To obtain a single **diversity score** for the datasets, we normalize the effective rank and participation ratio with the embedding dimension and compute their geometric mean. We observe that **FineVision is not only the biggest, but also the most diverse dataset**. Additionally, you can also clearly see that more images do not necessarily result in more diversity, since LLaVa is substantially less diverse than the Cauldron, even with more images.
 
 | Name          | Images | Effective Rank	| Participation Ratio 	| Diversity	|
 |---------------|--------|------------		|----------		          |---------- |
@@ -369,19 +377,19 @@ Similarly to the comparison of the size, we also wanted to evaluate the datasets
 | FineVision    | 17.3M  | 359.22		      | 182.52		            | 0.500     |
 
 ### Should you merge multiple questions for the same image into a single multi turn conversation?
-Since the training of a VLM already builds upon pretrained vision and language backbones, datasets are usually not completely unstructured, but follow an image+question and answer structure. Recent works have shown that consolidating multiple questions for the same image into a multi-turn conversation where the image is shown only once improves model performance, reduces training budget, and reduces the datasets’ memory footprint. We therefore experiment with deduplicating every image in our dataset internally using the same SSCD descriptors, manually inspect the resulting clusters, and merge fitting samples into a multi-turn conversation.
+Since the training of a VLM already builds upon pretrained vision and language backbones, datasets are usually not completely unstructured, but follow an image+question and answer structure. Some works have shown that consolidating multiple questions for the same image into a multi-turn conversation where the image is shown only once improves model performance, reduces training budget, and reduces the datasets’ memory footprint. We therefore experiment with deduplicating every image in our dataset internally using the same SSCD descriptors, manually inspect the resulting clusters, and merge fitting samples into a multi-turn conversation.
 
-When training with the same training budget, we find that both models perform very similarly. Some benchmarks favor one image/several turns, while others favor one image/one turn. Given this, we decide to release the dataset without merging multiple questions for the same image, and open-source the pipeline in case users want to explore this further.
+When training with the same training budget, we find that both models perform very similarly <a href="#internal-deduplication">(Fig. 5)</a>. Some benchmarks favor one image/several turns, while others favor one image/one turn. Given this, we decide to release the dataset without merging multiple questions for the same image, and open-source the pipeline in case users want to explore this further.
 
 ---
-<HtmlEmbed src="internal-deduplication.html" desc="Figure 5: Average Ranking of Models trained with internally deduplicated / merged samples. No clear benefit in merging ca be seen with respect to model performance." />
+<HtmlEmbed id="internal-deduplication" src="internal-deduplication.html" desc="Figure 5: Average Ranking of Models trained with internally deduplicated / merged samples. No clear benefit in merging can be seen with respect to model performance." />
 ---
 
 ### Should you train on multilingual data if your language backbone was not?
-There are some multilingual datasets in our mixture, but since our Language Backbone is only trained on English data, we experimented with removing all the multilingual, mainly Chinese, subsets. Our results show that there is a slight advantage in leaving the multilingual data, even if it was not part of the Language Backbone's initial training. We believe this reinforces our hypothesis that more diversity in the dataset is generally preferable for VLM training. In our training setup with this configuration, one epoch over the whole non-deduplicated dataset equals ~12k steps, so the benefit of unseen languages only materializes after the first full epoch.
+There are some multilingual datasets in our mixture, but since our Language Backbone is only trained on English data, we experimented with removing all the multilingual, mainly Chinese, subsets. Our results show that there is a slight advantage in leaving the multilingual data, even if it was not part of the Language Backbone's initial training. We believe this reinforces our hypothesis that more diversity in the dataset is generally preferable for VLM training. In our training setup with this configuration, one epoch over the whole non-deduplicated dataset equals ~12k steps, so the benefit of unseen languages only materializes after the first full epoch <a href="#remove-ch">(Fig. 5)</a>.
 
 ---
-<HtmlEmbed src="remove-ch.html" desc="Figure 6: Average Rank of Models trained with and without multilingual samples. Keeping samples in unseen langauges improves performance after the first epoch." />
+<HtmlEmbed id="remove-ch" src="remove-ch.html" desc="Figure 6: Average Rank of Models trained with and without multilingual samples. Keeping samples in unseen langauges improves performance after the first epoch." />
 ---
 
 ### How can you assess the quality of the dataset?
@@ -390,29 +398,31 @@ The usual goal for every dataset, to collect samples with the highest quality po
 To try to quantify the quality of the training data and the effect it has on the model’s performance, we run extensive ablations on our generated ratings.
 
 ---
-<HtmlEmbed src="all-ratings.html" desc="Figure 6: Average Rank of Models trained with samples that have all 4 ratings above a certain threshold. Keeping all samples results in the best performance." />
+<HtmlEmbed id="all-ratings" src="all-ratings.html" desc="Figure 6: Average Rank of Models trained with samples that have all 4 ratings above a certain threshold. Keeping all samples results in the best performance." />
 ---
 
-Interestingly, both when only training on turns that have any of the 4 ratings  under a certain threshold, as well as when training on turns where only a single rating at a time is used, we observe the same behaviour. Simply training on the most diverse data, that one containing all samples, outperforms in benchmarks. This could mean multiple things. 
-Firstly, we can see almost the same distribution in the ranks across all filters: From best to worst with an increase in the rating threshold. For example the visual dependency and the image correspondence rating both result in exactly the same distribution of rankings, corresponding to the natural order of options, 1 through 5. This could indicate that with a sufficiently large dataset that you train on long enough, it hurts more to remove samples, even if they were judged to be of low quality, than to train on them. 
+Interestingly, both when only training on turns that have any of the 4 ratings  under a certain threshold, as well as when training on turns where only a single rating at a time is used, we observe the same behaviour. Simply training on the most diverse data, that one containing all samples, outperforms in benchmarks <a href="#all-ratings">(Fig. 6)</a> <a href="#all-filters">(Fig. 7)</a>. This could mean multiple things. 
+Firstly, we can see almost the same distribution in the ranks across all filters: from best to worst with an increase in the rating threshold. For example the visual dependency and the image correspondence rating both result in exactly the same distribution of rankings, corresponding to the natural order of options, 1 through 5. This could indicate that with a sufficiently large dataset that you train on long enough, it hurts more to remove samples, even if they were judged to be of low quality, than to train on them. 
 
-Additionally, the notion of quality for VLM datasets is nuanced in general. If we compare training a VLM and an LLM, training the VLM is closer in nature to the SFT part than the ‘Pre-Training’ part of training a LLM. We do not train on crawls of internet data, instead we train on individual samples of Image-Question and Answer pairs, and these datapoints are usually ‘curated rather than collected’. We also do not train on trillions of tokens, but on billions. This means that the datasets for VLMs usually already have a certain baseline quality. Since FineVision is mainly a collection of common VLM datasets, combined with a few newly created ones in low resource domains, this baseline quality is the same here. We could therefore be trying to measure and quantify nuances in the quality of Image-Question-Answer Pairs, instead of using the binary indicator of using curated SFT datasets as the measure for quality, and training on as much data as possible.
+Additionally, the notion of quality in VLM datasets is inherently nuanced. Unlike LLMs, where pre-training often relies on massive web crawls, training a VLM is closer to the supervised fine-tuning (SFT) stage. We do not train on crawls of internet data, instead we train on individual samples of Image-Question and Answer pairs, and these datapoints are usually ‘curated rather than collected’. We also do not train on trillions of tokens, but on billions. This built-in curation provides a baseline level of quality from the start. FineVision follows this pattern: it brings together widely used VLM datasets along with a few new ones in low-resource domains. We could therefore be trying to measure and quantify noisy nuances in the quality of Image-Question-Answer Pairs, instead of using the fact that they are already curated SFT datasets as the measure for quality.
 
 Alternatively, while we used state-of-the-art open source models to judge our datapoints, we still had to find a compromise between model quality and cost due to the raw required effort to rate every single turn of FineVision. The chosen models could simply not be powerful enough to recognize and judge the quality of samples.
 Even though our first proposal to judge the quality of multimodal data on a per-turn basis did not yield any improvement in model performance, we believe that this is still an exciting and important direction of research and hope the release of FineVision encourages the community to develop techniques for this at large scale.
 
+---
 <Wide>
-<HtmlEmbed src="filters-quad.html" title="Model Performance After Applying Individual Filters" desc="Figure 7: Comparison across thresholds for all four filters individually: Formatting, Relevance, Visual Dependency, and Image-Question Correspondence." align="center" />
+<HtmlEmbed id="all-filters" src="filters-quad.html" title="Model Performance After Applying Individual Filters" desc="Figure 7: Comparison across thresholds for all four filters individually: Formatting, Relevance, Visual Dependency, and Image-Question Correspondence. Keeping all samples results in the best average performance." align="center" />
 </Wide>
+---
 
 ### Should you train in multiple stages?
 The standard training procedure of a VLM usually follows at least two stages. First, you train only the connecting module, potentially in addition the image encoder, and then you train the whole model in a second stage. Some work has even introduced an additional Stage 2.5, where you train the full model on a smaller subset of higher quality data. To investigate this on small models, we experiment both with single, dual and triple stage training.
 
 ---
 #### 1 Stage vs 2 Stages
-To evaluate if pre-training the Modality Projection and the Vision Encoder provides any benefits to the final model performance, we conduct this experiment at a higher image resolution of 2048px and train substantially longer. We can see that even for training longer, the general difference in model performance is quite small. Individual benchmarks, do show differences (ScienceQA drops by 5% but OCRBench improves by 5% in the two-stage setup), so the better setup is individual to the desired model capabilities. This also shows that evaluation (and through this also selecting a training procedure) a VLM is not straightforward tasks, since availible benchmarks are limited proxies for the underlying model performance.
+To evaluate if pre-training the Modality Projection and the Vision Encoder provides any benefits to the final model performance, we conduct this experiment at a higher image resolution of 2048px and train substantially longer. We can see that even for training longer, the general difference in model performance is quite small. Individual benchmarks, do show differences (ScienceQA drops by 5% but OCRBench improves by 5% in the two-stage setup) <a href="#ss-vs-s1">(Fig. 8)</a>, so the better setup is individual to the desired model capabilities. This also shows that evaluation (and through this also correctly training) a VLM is not straightforward tasks, since availible benchmarks are limited proxies for the underlying model performance.
 
-<HtmlEmbed src="ss-vs-s1.html" desc="Figure 8: Average Rank of a model trained for 60K steps in a single stage, and a model trained for the same 60k steps on top of pretraining the Modality Projection and Vision Encoder for 15k steps. The pre-training procedure is not depicted in this graph." />
+<HtmlEmbed id="ss-vs-s1" src="ss-vs-s1.html" desc="Figure 8: Average Rank of a model trained for 60K steps in a single stage, and a model trained for the same 60k steps on top of pretraining the Modality Projection and Vision Encoder for 15k steps. The pre-training procedure is not depicted in this graph." />
 
 ---
 #### 2 Stages vs 2.5 Stages
@@ -420,9 +430,9 @@ We also experiment if splitting the second stage results in any performance impr
 
 We take the baseline, and continue training for another 20k steps, both with the unfiltered (>= 1) as well as filtered subsets of **FineVision** according to our ratings.
 
-<HtmlEmbed src="s25-ratings.html" desc="Figure 9: Average Rank if a model trained for an additional 20K steps on top of unfiltered training for 20K steps. Subselecting data for the final training steps does not yield a performance improvement with our quality measure." />
+<HtmlEmbed id="s25-ratings" src="s25-ratings.html" desc="Figure 9: Average Rank if a model trained for an additional 20K steps on top of unfiltered training for 20K steps. Subselecting data for the final training steps does not yield a performance improvement with our quality measure. Only the 20k steps for the final stage are depicted here, the first 20k steps are the same for all variations." />
 
-As in the previous results, we observe that the best outcome is simply achieved by training on as much and as diverse data as possible. Like before, this could also be due to the way we filter the data, and a different quality measure might yield different results.
+As in the previous results, we observe that the best outcome is simply achieved by training on as much and as diverse data as possible <a href="#s25-ratings">(Fig. 9)</a>. Like before, this could also be due to the way we filter the data, and a different quality measure might yield different results.
 
 ## Conclusion
 We introduce **FineVision**, a new state of the art open dataset to train VLMs, that is both **bigger and more diverse** than previous open source datasets. We provide extensive analysis regarding size, diversity, contamination and data-centric model training, and hope we can empower both further research and the community with this.

app/src/content/bibliography.bib

@@ -2080,3 +2080,13 @@
 	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning},
 	file = {Preprint PDF:/Users/luis/Zotero/storage/5JNXNX76/Toshniwal et al. - 2024 - OpenMathInstruct-2 Accelerating AI for Math with Massive Open-Source Instruction Data.pdf:application/pdf},
 }
+
+@misc{weyand2020googlelandmarksdatasetv2,
+      title={Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval}, 
+      author={Tobias Weyand and Andre Araujo and Bingyi Cao and Jack Sim},
+      year={2020},
+      eprint={2004.01804},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2004.01804}, 
+}

app/src/content/embeds/against-baselines-deduplicated.html

@@ -230,7 +230,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/against-baselines.html

@@ -220,7 +220,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/all-ratings.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/formatting-filters.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/image-correspondence-filters.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/internal-deduplication.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/relevance-filters.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/remove-ch.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/s25-ratings.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/ss-vs-s1.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',

app/src/content/embeds/visual-dependency-filters.html

@@ -219,7 +219,7 @@
         'ocrbench_ocrbench_accuracy': 'OCRBench',
         'scienceqa_exact_match': 'ScienceQA',
         'textvqa_val_exact_match': 'TextVQA',
-        'average': 'Average (excl. MME)',
+        'average': 'Average',
         'average_rank': 'Average Rank',
         'ai2d_exact_match': 'AI2D',
         'chartqa_relaxed_overall': 'ChartQA',