## ARE BIGGER ENCODERS ALWAYS BETTER IN VISION LARGE MODELS?

A PREPRINT

Bozhou Li , Hao Liang , Zimo Meng , Wentao Zhang † † † Peking University

August 2, 2024

## ABSTRACT

In recent years, multimodal large language models (MLLMs) have shown strong potential in real-world applications. They are developing rapidly due to their remarkable ability to comprehend multimodal information and their inherent powerful cognitive and reasoning capabilities. Among MLLMs, vision language models (VLM) stand out for their ability to understand vision information. However, the scaling trend of VLMs under the current mainstream paradigm has not been extensively studied. Whether we can achieve better performance by training even larger models is still unclear. To address this issue, we conducted experiments on the pretraining stage of MLLMs. We conduct our experiment using different encoder sizes and large language model (LLM) sizes. Our findings indicate that merely increasing the size of encoders does not necessarily enhance the performance of VLMs. Moreover, we analyzed the effects of LLM backbone parameter size and data quality on the pretraining outcomes. Additionally, we explored the differences in scaling laws between LLMs and VLMs.

## 1 Introduction

In recent years, the rapid development of large language models has revolutionized the field of natural language processing Devlin et al. [2019], Touvron et al. [2023], Achiam et al. [2023], Bai et al. [2023], Du et al. [2022]. These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation Fan et al. [2021], question answering Devlin et al. [2019], Raffel et al. [2020], and text generation Brown et al. [2020], Du et al. [2022]. Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities Achiam et al. [2023], Chu et al. [2024], Maaz et al. [2023].

Among MLLMs, VLMs achieve competitive performance in traditional multimodal tasks such as image classification Chen et al. [2024], image understanding Li et al. [2023], and image captioning Bai et al. [2023]. Moreover, their excellent language understanding capabilities enable strong performance in text-rich tasks, such as vision questionanswering Liu et al. [2024a,b] and image-text retrieval Chen et al. [2024].

Currently, the dominant architecture for VLMs employs a large language model as the backbone. Mechanisms such as cross-attention Li et al. [2023] or linear projectors Liu et al. [2024a] are utilized to connect the vision encoder (e.g., ViT Dosovitskiy et al. [2020]) with the LLM backbone Zhang et al. [2024a]. Larger visual encoders possess stronger encoding capabilities, allowing them to extract features more effectively. These features are then transformed by the projectors for the LLMs to understand. We refer to this model architecture as the connected vision paradigm.

Apart from VLMs, previous research on scaling laws Kaplan et al. [2020], Hoffmann et al. [2022] adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data. Under the assumption that the scaling trend follows a power-law relationship, researchers fit power-law formulas by training

† Equal Contribution

models of different sizes with varying amounts of data. Additionally, scaling laws have been extended to various scenarios, including graph neural networks Liu et al. [2024c], data mixing Ye et al. [2024], data pruning Sorscher et al. [2022], and fine-tuning LLMs Zhang et al. [2024b]. Although Aghajanyan et al. [2023] investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm Zhang et al. [2024a], Bai et al. [2023], leading to the following challenges:

- C1. Poor Effectiveness: Since no scaling laws have been conducted, it is uncertain whether this paradigm can scale up to achieve better performance, resulting in suboptimal model performance.
- C2. Low Efficiency: Without scaling laws, we do not know how much data or how many parameters to use. Typically, all available data are used for training VLMs, resulting in a waste of data and computational resources.

To address these issues, we conduct scaling law experiments following Kaplan et al. [2020]. We select the renowned LLaVA1.5 Liu et al. [2024b] model as the backbone of our study. To investigate the scaling laws, we utilize models with 7 billion and 13 billion parameters. For the dataset, we choose image-text pairs from CC12M Changpinyo et al. [2021] and Laion400M Schuhmann et al. [2021], known for their high-quality pairs. We use data sizes from 1 million to 10 million image-text pairs for our scaling law experiments.

The core contributions of this paper are summarized as follows:

- · New Perspective To the best of our knowledge, we are the first to conduct scaling law studies for the connect vision paradigm. Additionally, we are the first to analyze the limitations of this VLM paradigm.
- · New Observation We conducted experiments on the pretraining phase of MLLMs using the connect vision paradigm. From the experimental results, we derived a crucial conclusion: simply utilizing a ViT trained using CLIP with more parameters and better performance does not enhance the performance of MLLMs. This suggests that exploring alternative methods is necessary to improve the performance of MLLMs. We also indicate that this issue is highly likely not caused by limitations in the scaling ability of ViT.

## 2 Background and Related Work

## 2.1 Multimodal Large Language Model

In recent years, benefiting from the development of model architectures based on transformers Vaswani et al. [2017] and the increase in computing resources and training data, LLMs Devlin et al. [2019], Touvron et al. [2023], Achiam et al. [2023], Bai et al. [2023], Du et al. [2022] have emerged. These models possess massive parameters, requiring extensive computational resources to train on large datasets for a prolonged period. The sheer scale of these models, often exceeding billions of parameters, has enabled them to capture subtle nuances of language and learn complex patterns from vast amounts of text data. LLMs have not only demonstrated outstanding performance in traditional NLP tasks but have also exhibited emergent capabilities such as reasoning, code generation, and solving mathematical problems Wei et al. [2022].

With the fast-paced advancement of LLMs, a natural question arises: how can we enhance the capabilities of LLMs to enable them to understand information from other modalities? Some earlier MLLMs, such as BEiT-3, attempted to train a multimodal model from scratch using data from multiple modalities Wang et al. [2022]. However, there is now a growing preference for combining pre-trained LLMs and modality encoders using specific mechanisms to construct multimodal LLMs. The methods combining pre-trained LLMs and modality encoders can be classified into four categories Wadekar et al. [2024]. The first and second categories employ cross-modal attention mechanisms Alayrac et al. [2022] and custom layers Cho et al. [2021], respectively, to integrate information from other modalities into the internal representation of the LLM. These two types of methods are commonly referred to as deep fusion methods. The third category uses different modules to align information from other modalities with text information Liu et al. [2024a], Li et al. [2023], Bai et al. [2023]. Researchers exploring the fourth category attempt to unify the encoding of different modalities and expand the vocabulary of LLMs Zhan et al. [2024]. These two methods are commonly referred to as early fusion methods. Due to their ease of training, high computational efficiency, and ability to construct any-to-any MLLMs, early fusion methods, especially the third category, which is also the focus of this article, are gradually becoming mainstream.

In general, the fusion modules used in the third category of methods can be classified into three types. The first type is transformer-based abstractors. These methods allow for adjusting the number of visual tokens to balance model performance and efficiency Li et al. [2023], Bai et al. [2023]. However, they might lose locality inductive bias and require longer training times and more training data. The second type involves using MLPs (Multi-Layer Perceptrons) Liu et al. [2024a]. This approach is lightweight, easy to train, and capable of capturing local information. However, it

can only generate fixed-length visual tokens. Furthermore, some researchers have explored the use of custom layers to combine the advantages of both approaches Cha et al. [2024].

The training process of MLLMs can be divided into two stages: multimodal pretraining and multimodal instruction finetuning. During the multimodal pretraining (MM PT) stage, the input and output projectors are trained to align different modalities by optimizing predefined objectives. Typically, X-text pairs are used in this stage, where X represents data from other modalities. In the multimodal instruction fine-tuning (MM IT) stage, the pre-trained multimodal LLM is fine-tuned using a dataset formatted in an instructional manner. This fine-tuning aims to enhance the model's generalization ability on unseen tasks. In this article, we conducted experiments on the MM PT stage.

## 2.2 Scaling Law

In the process of deep learning development, there has been a tendency to train larger models to achieve better performance. A natural question arises: as the number of model parameters and the amount of training data increase, how does model performance change? This has led to research on scaling laws. Hestness et al. Hestness et al. [2017] were the pioneers in using empirical methods to study the scaling law of deep neural networks. Kaplan et al. Kaplan et al. [2020] and Hoffmann et al. Hoffmann et al. [2022] conducted extensive experiments on transformer-based autoregressive models at different scales. They investigated the relationship between model loss and the number of model parameters and training data.

In situations where the model architecture is known, we can approximate the training cost based on the number of model parameters and the amount of training data (e.g., C ≈ 6 ND for the transformer-based model, where N represents the number of model parameters, D represents the amount of training data, and C represents the computational costs). By combining this estimation with scaling laws, we can determine the optimal allocation of training data and model parameters within limited computational resources Hoffmann et al. [2022]. This approach is particularly useful for LLMs since training them requires a significant amount of computational resources. Even if we embrace the philosophy underlying LLAMA Touvron et al. [2023], which entails setting aside computational expenses during the training phase and maximizing data usage to augment model performance at inference time, we consequently face an additional consideration: using larger models inherently translates to increased computational costs during inference. Therefore, even under conditions of ample data availability, a critical question arises: is the performance boost from expanding model parameter counts justified by the resultant higher computational overhead at inference? Under this assumption, scaling laws are still necessary.

After the initial scaling law was proposed, researchers studied the scaling law in different scenarios, such as graph neural networks Liu et al. [2024c], data mixing Ye et al. [2024], data pruning Sorscher et al. [2022], and fine-tuning of large language models Zhang et al. [2024b]. Some of them made certain adjustments to the form of the formula.

When it comes to MLLMs, Aghajanyan et al. Aghajanyan et al. [2023] explored the relationship between the loss and the dataset size and the parameter size over seven modalities, including text, image, image-text, speech, speech-text, code, and molecules. They also examined the scaling law when models are pre-trained on two modalities simultaneously and modified the form of the formula. However, the training paradigm they used in their study differs from the connected vision paradigm, the mainstream methods preferred today. They trained LLMs and modality encoders from scratch, similar to the approach chosen by BEiT-3 Wang et al. [2022], to obtain their scaling law. This discrepancy has sparked our research focus. To the best of our knowledge, the scaling law of multimodal large models under the connected vision paradigm is still under-researched.

## 3 Methods

## 3.1 Dataset

Weutilized the CC12M dataset Changpinyo et al. [2021] in our experiments, a comprehensive and expansive multimodal collection designed to foster advancements in artificial intelligence, particularly focusing on the intersection of vision and language. From the CC12M dataset, we strategically extracted subsets of data ranging from 1 million to 10 million images, incrementing by 1 million each time, to serve as our training datasets. This approach was chosen to systematically investigate the impact of varying training data sizes on the model's performance and learning outcomes. In our experiments involving different variants and scales of the Vision Transformer (ViT) trained by CLIP Radford et al. [2021], we maintained the use of identical datasets for each training session. This decision was made to eliminate the influence of random variables that could arise from using different data splits. By controlling for the input data, we aimed to isolate and observe the effects of model architecture and size on the experimental results, providing a clearer understanding of how these factors influence performance.

To validate the generality of our findings, we also conducted a set of experiments using the LAION-400M dataset Schuhmann et al. [2021]. Similarly, we constructed training datasets ranging from 1M to 10M by randomly sampling from the LAION-400M dataset.

For each image-text pair in the dataset, we randomly selected a text prompt from the LLaV A paper's text prompts Liu et al. [2024a] and randomly arranged its order with the image. We used the caption of the image as the ground truth. For each size of the training dataset, we extracted 25,600 of them as validation sets.

## 3.2 Model Choice

In our quest to elucidate how different sizes of the ViT influence the performance of multimodal large language models, we deemed it crucial to employ ViTs of various sizes. To ensure that extraneous factors such as disparities in training data distribution or variations in training recipes did not skew our findings, we meticulously opted for a consistent series of ViT models throughout our experiments.

Our choice fell upon the ViT models Ilharco et al. [2021] trained using CLIP on the extensive Laion2B Schuhmann et al. [2022] dataset, renowned for its comprehensive coverage and quality, thereby providing a robust baseline for our investigations. The specific model sizes we selected are shown in Table 1.

Table 1: Parameter sizes of various CLIP models and their ViT modules from the LAION dataset.

| Model Name                            | Parameter Size of CLIP Model   | Parameter Size of ViT Modules   |
|---------------------------------------|--------------------------------|---------------------------------|
| laion/CLIP-ViT-B-16-laion2B-s34B-b88K | 150M                           | 86M                             |
| laion/CLIP-ViT-L-14-laion2B-s32B-b82K | 428M                           | 304M                            |
| laion/CLIP-ViT-H-14-laion2B-s32B-b79K | 986M                           | 632M                            |
| laion/CLIP-ViT-g-14-laion2B-s34B-b88K | 1.37B                          | 1.01B                           |

For the LLM backbone, the Vicuna series Chiang et al. [2023] was selected. This model series was cultivated by fine-tuning the LLaMA2 Touvron et al. [2023] foundational model on a diverse set of user-shared dialogues sourced from ShareGPT Sha. To ensure the reliability of our conclusions, we conducted training on both the 7B model and the 13B model from the Vicuna series.

## 3.3 Training Process

In our overarching approach, we adopted the training process from the MM PT stage of LLaVA v1.5 Liu et al. [2024b]. Specifically, a two-layer Multi-Layer Perceptron (MLP) served as the linear projector within our framework, utilizing the Gaussian Error Linear Units (GELU) Hendrycks and Gimpel [2016] as the activation function to enhance non-linearity. Throughout the training phase, we froze the parameters of the ViT and LLM backbone and only updated the Linear Projector.

We conducted the MM PT stage of the model using next token prediction as the training task and cross-entropy as the loss function. During the training process, we only masked the textual data in the ground truth and did not mask the image tokens.

Across a spectrum of dataset sizes, ranging from smaller to larger volumes, and varying the scale of ViT parameters and the size of the Vicuna model parameters, we trained the linear projector from scratch with a cosine scheduler. All experiments were conducted on 8*A100/A800 NVIDIA GPU machines.

## 4 Experiments Results

Tables 2 and 3 exhibit the concluding evaluation loss values obtained when employing various dataset sizes and ViT model sizes and training datasets built from CC12M. To render these outcomes more comprehensible at a glance, we have also produced Figures 1 and 2. These visual aids aim to depict the relationship between model size, dataset size, and training loss, making the trends and patterns easier to discern. Table 4 and Figure 3 show the results with datasets built from LAION-400M.

Drawing insights from our experimental findings, several noteworthy observations emerge:

## · Increasing Data Quantity Improves Model Performance

It is observed that for smaller training datasets, augmenting the data quantity effectively reduces the evaluation loss. This aligns with the learning theory that more examples lead to better model performance.

## · Larger LLM Backbone Enhances Model Performance

Comparing the training outcomes on the Vicuna-7B and Vicuna-13B models reveals that larger models consistently present lower evaluation loss than their smaller versions. This implies that larger language models possess a heightened capability to understand and interpret vision tokens.

## · The Importance of High-Quality Data

The evaluation loss obtained from training on the CC12M dataset is lower than the evaluation loss from training on the LAION-400M dataset. The ViT models used in the experiment were trained on the LAION-2B dataset. Intuitively, its distribution should be closer to LAION-400M, and training multimodal LLMs with LAION-400M would yield better results. However, experimental results show that using CC12M is more effective. This is likely due to the quality of the datasets. Compared to LAION-400M, the CC12M dataset has higher quality, with a higher degree of matching between image and text. This result emphasizes the importance of high-quality datasets when training MLLMs.

## · Larger LLM Backbone Requires Less Training Data

Notably, when leveraging the Vicuna-13B model, the evaluation loss plateaus around the 7M mark in training data size, signifying a diminishing return on additional data. Conversely, the Vicuna-7B model continues to benefit from an increased data volume, illustrating a more gradual learning curve. This indicates that smaller models might require more substantial data increments to observe subsequent improvements, while larger models can exhibit more pronounced progress with less data.

## · Improved ViT Performance Doesn't Guarantee Better Results

Acritical finding from our study reveals that merely amplifying the parameter scale of ViT does not necessarily translate into enhanced model performance. Although vision transformers exhibit a relatively lesser scaling capacity compared with text transformers Zhai et al. [2022], the parameter scales selected in our experiments are significantly removed from the upper limits of ViT's scaling capabilities.

The 632M ViT trained using CLIP achieves a zero-shot accuracy of 78.0% on the ImageNet-1K dataset Russakovsky et al. [2015] while increasing the model parameter size to 1.01B and training on a larger amount of data results in a zero-shot accuracy leap to 78.4% Ilharco et al. [2021]. Moreover, through architectural advancements, researchers have demonstrated ViT models boasting accuracies above 90%, with parameter counts around 20B Zhai et al. [2022], Dehghani et al. [2023]. This underscores that within our experimental parameters, ViT's scaling potential has not been saturated, with enhanced performance still achievable through augmented model dimensions.

However, within our specific scenario, increasing the ViT's size does not yield superior results and, in certain instances, may lead to performance degradation. When training with the CC12M dataset, the validation loss of multimodal LLMs constructed with the 1.01B CLIP model's ViT is lower than that of models using the 632M model only when the training data volume is large. However, when training with data from the LAION-400M dataset, the validation loss of models using the 1.01B model consistently remains lower than that of models using the 632M model. This suggests that additional factors are at play, hindering the alignment and effectiveness between ViT and LLM in MM PT stage, necessitating deeper investigation.

Table 2: Evaluation Loss Using Vicuna-7B as LLM Backbone, Trained on Data Sampled from CC12M

| Parameter Size   |    1M |    2M |    3M |    4M |    5M |    6M |    7M |    8M |    9M |   10M |
|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| 86M              | 2.168 | 2.12  | 2.088 | 2.066 | 2.051 | 2.062 | 2.055 | 2.033 | 2.04  | 2.023 |
| 304M             | 2.056 | 2.013 | 1.983 | 1.962 | 1.946 | 1.947 | 1.953 | 1.929 | 1.942 | 1.932 |
| 632M             | 2.036 | 1.987 | 1.96  | 1.937 | 1.926 | 1.921 | 1.931 | 1.896 | 1.915 | 1.886 |
| 1.01B            | 2.017 | 1.987 | 1.952 | 1.931 | 1.926 | 1.936 | 1.926 | 1.919 | 1.922 | 1.909 |

Table 3: Evaluation Loss Using Vicuna-13B as LLM Backbone, Trained on Data Sampled from CC12M

| Parameter Size   |    1M |    2M |    3M |    4M |    5M |    6M |    7M |    8M |    9M |   10M |
|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| 86M              | 2.072 | 2.036 | 2.014 | 1.98  | 1.962 | 1.966 | 1.947 | 1.946 | 1.956 | 1.937 |
| 304M             | 1.977 | 1.933 | 1.903 | 1.88  | 1.867 | 1.865 | 1.85  | 1.852 | 1.868 | 1.839 |
| 632M             | 1.96  | 1.907 | 1.884 | 1.864 | 1.846 | 1.848 | 1.819 | 1.821 | 1.833 | 1.814 |
| 1.01B            | 1.936 | 1.895 | 1.883 | 1.876 | 1.862 | 1.851 | 1.845 | 1.841 | 1.843 | 1.82  |

Table 4: Evaluation Loss Using Vicuna-7B as LLM Backbone, Trained on Data Sampled from LAION-400M

| Parameter Size   |    1M |    2M |    3M |    4M |    5M |    6M |    7M |    8M |    9M |   10M |
|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| 86M              | 2.309 | 2.289 | 2.246 | 2.214 | 2.23  | 2.218 | 2.218 | 2.198 | 2.196 | 2.177 |
| 304M             | 2.169 | 2.148 | 2.104 | 2.072 | 2.086 | 2.081 | 2.075 | 2.063 | 2.056 | 2.039 |
| 632M             | 2.135 | 2.122 | 2.07  | 2.037 | 2.051 | 2.036 | 2.047 | 2.032 | 2.019 | 2.012 |
| 1.01B            | 2.144 | 2.127 | 2.088 | 2.057 | 2.076 | 2.065 | 2.059 | 2.042 | 2.037 | 2.022 |

Figure 1: Using Vicuna-7B as LLM backbone, trained on data sampled from CC12M

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00620v1-figure-1c746f64-3de1-4c14-999c-e77babb7dc70.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00620v1-figure-5a0b376e-0553-492a-977a-5890cbf4f783.jpg)

Figure 2: Using Vicuna-13B as LLM backbone, trained on data sampled from CC12M

Figure 3: Using Vicuna-7B as LLM backbone, trained on data sampled from LAION-400M

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00620v1-figure-1b4e758d-b8c4-4897-8fde-a9f332257463.jpg)

## 5 Discussion &amp; Conclusion

Our experimental outcomes underscore a critical observation: within the LLaVA Liu et al. [2024a] framework, the simplistic augmentation of ViT's parameter scale does not correspondingly elevate the performance of MLLMs. This finding necessitates a strategic shift in our approaches to multimodal model design and optimization.

## · Strengthen Data Alignment Across Modalities With Data-centric Approaches

The experimental insights suggest the importance of improving the quality of training data for the alignment of different modalities. In this pursuit, data-centric approaches appear to hold significant promise. By focusing on enhancing data quality, variety, and relevance, we can potentially achieve better alignment between visual and textual modalities with higher efficiency, potentially leading to superior performance in multimodal applications Bai et al. [2024].

## · Investigate The Differences Between Various Alignment Methods

Deep Dive into Underlying Performance Limitations: A comprehensive investigation into why ViT performance enhancements do not translate into superior multimodal model performance is warranted. This exploration should encompass the dynamics of data utilization, model architecture, and training methodologies. When comparing the training approaches of CLIP Radford et al. [2021] and LLaVA Liu et al. [2024a] we can observe certain differences in alignment strategies. The popular CLIP model utilizes a decoder-only LLM as the text encoder, such as GPT-2 Radford et al.. In CLIP, the CLS token of the image processed by ViT is

aligned with the EOS token of the text processed by GPT-2. In contrast, the processing approach of LLaV A is more akin to mapping the image tokens processed by ViT to the embedding space of a LLM using MLP layers. This differs from aligning image tokens with text tokens processed by GPT-2, as done in CLIP. The impact of these alignment method differences on model performance is worth exploring.

## · Exploration of Multimodal Information Fusion

- - Architectural Innovations for Multimodal Integration: There is a need to explore novel architectures that facilitate more seamless integration of multimodal information Cha et al. [2024], Li et al. [2023]. This could involve designing frameworks that better enable the exchange and processing of visual, textual, and other modal data, potentially leading to enhanced overall performance.
- -Expansion of Vocabulary: Considering the inclusion of techniques like AnyGPT Zhan et al. [2024] to extend the vocabulary can prove beneficial in enriching the model's understanding and processing capabilities across different modalities. This could potentially enable the model to capture and utilize a broader spectrum of information, thereby enhancing its performance.

The observed phenomenon of larger LLMs requiring smaller datasets for alignment with ViT suggests a closer alignment between the semantic spaces of vision and text. This insight not only enriches our understanding of multimodal model dynamics but also opens avenues for optimizing data requirements and training strategies for MLLMs. Future research should delve deeper into the mechanisms behind these observations, aiming to leverage the inherent semantic alignment capabilities of larger LLMs for enhanced performance and efficiency in multimodal applications.

## 6 Limitations and Future Work

Despite our study encompassing an extensive series of one hundred and twenty experiments across various parameter combinations, the investigation into the scaling laws of MLLMs during the MM PT stage remains a vast and complex field. Our current exploration, while insightful, is not exhaustive in addressing the comprehensive nature of this topic.

Compared to the scaling law of LLMs Kaplan et al. [2020], Hoffmann et al. [2022], the scaling law of MLLMs exhibits more complex characteristics.

## · More Influencing Factors

More factors influence the performance of MLLMs, particularly when using pre-trained modality encoders and an LLM backbone to build MLLMs, the paradigm that is currently widely used. The parameter size of modality encoders, the amount of data used for pretraining the modality encoders and LLM backbone, the quality of data during the MM PT stage, and the distribution discrepancy between the data used for MM PT and the data used for pretraining the modality encoders and LLM backbone can potentially influence model performance. These factors may also interact with each other in more complex ways. In addition, different modalities, fusion methods for integrating multimodal information, and architectures of modality encoders may exhibit different forms of scaling law.

## · Different Estimates of Computational Costs

Under the current training framework, it is not appropriate to solely rely on C ≈ 6 ND for estimating computational costs. Due to the utilization of the backpropagation algorithm to update the model parameters, even if we only update the parameters of the MLP layer, which is quite lightweight, we still need to compute gradients for a significant portion of the LLM parameters. In addition, the modality encoder only needs to process data from other modalities, while the LLM also needs to handle text tokens. Taking LLaVA as an example, the compute budget for the MM PT stage of LLaVA can be estimated as:

$$C \approx 2 N _ { V i T } D _ { i m g } + 6 N _ { L L M } ( D _ { i m g } + D _ { t x t } )$$

where N LLM and N V iT represent the number of parameters of LLM and ViT, and D img and D txt represent the number of image tokens and text tokens, respectively. The MLP layer is overlooked due to its relatively small number of parameters. It is worth noting that the value of D img is dependent on the architecture of ViT, while the ratio between D txt and D img is also dependent on the source of the image-text pairs. For other fusion methods and modality encoder architectures, this formula may have different forms. This introduces new challenges in making trade-offs between computational costs and model performance.

To firmly substantiate the conclusions drawn in this paper, a more extensive empirical foundation is required. This necessitates conducting additional experiments on a broader range of LLM families, encompassing diverse architectures and capabilities. Moreover, the exploration should be extended to incorporate a wider variety of datasets-both in terms

of content and scale-to ensure a comprehensive understanding of how multimodal models behave under different conditions.

Beyond empirical evidence, the surprising outcomes observed in our experiments demand a theoretical underpinning that goes beyond mere data analysis. The unexpected findings, such as the non-intuitive scaling behaviors of multimodal models or the peculiar interactions between vision and language semantic spaces, call for a deeper level of understanding that can only be achieved through rigorous theoretical analysis.

The prohibitive cost associated with extensive experimental setups has constrained our investigation to a specific scenario: the performance of large models where both ViT and LLM components are frozen. This approach, while providing valuable insights into the behavior of multimodal models under resource constraints, deviates significantly from the methodologies employed by some mainstream multimodal models such as Qwen-vl Bai et al. [2023].

Our current experimental endeavors have been confined to the LLaVA architecture, providing a focused exploration of scaling properties within this specific framework. However, the landscape of multimodal AI architectures extends far beyond LLaVA, encompassing a diverse array of designs that include transformer-based connector architectures Li et al. [2023] and V AE-based vision encoder architectures Kim et al. [2021]. Each of these architectural paradigms embodies unique characteristics and potential scaling behaviors that require systematic investigation.

Our experiments only considered MLLMs that can understand different modalities but can only generate textual information. We did not take into account more powerful yet complex MLLMs that can generate content in other modalities. The study of the scaling behavior of such models is more complex and requires additional computational resources.

## References

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, 2019.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547-5569. PMLR, 2022.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research , 22(107):1-48, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 , 2024.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 , 2023.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24185-24198, 2024.

| Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.                                                          |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024a.                                                                                                                                                       |
| Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024b.                                                                                                                                                       |
| Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. |
| Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 , 2024a.                                                                                                               |
| Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.                                                                             |
| Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.                                |
| Jingzhe Liu, Haitao Mao, Zhikai Chen, Tong Zhao, Neil Shah, and Jiliang Tang. Neural scaling laws on graphs. arXiv preprint arXiv:2402.02054 , 2024c.                                                                                                                                                    |
| Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952 , 2024.                                                                                            |
| Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems , 35:19523-19536, 2022.                                                                         |
| Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193 , 2024b.                                                                                                                    |
| Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning , pages 265-279. PMLR, 2023.           |
| Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3558-3568, 2021.                           |
| Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.                                 |
| Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems , 2017. URL https: //api.semanticscholar.org/CorpusID:13756489 .                                    |
| Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 2022.                                                                   |
| Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.                   |
| Shakti N Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello. The evolution of multimodal model architectures. arXiv preprint arXiv:2405.17927 , 2024.                                                                                                                                     |
| Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.      |
| Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , pages 1931-1942. PMLR, 2021.                                                                                                                   |

| Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226 2024.                                                                                                      |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13817-13827, 2024.                                                                                                   |
| Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, MdMostofa Ali Pat- wary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 2017.                                                                                              |
| Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.                                                            |
| Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773 . If you use this software, please cite it as below. |
| Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.    |
| Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6, 2023.                                      |
| sharegpt. https://sharegpt.com/.                                                                                                                                                                                                                                                                                                        |
| Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.                                                                                                                                                                                                                            |
| Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12104-12113, 2022.                                                                                                                         |
| Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.                                                               |
| Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning , pages 7480-7512. PMLR, 2023.               |
| Tianyi Bai, Hao Liang, Binwang Wan, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Conghui He, Binhang Yuan, and Wentao Zhang. A survey of multimodal large language model from a data-centric perspective. arXiv preprint arXiv:2405.16640 , 2024.                                                                                         |
| Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.                                                                                                                                                                                            |
| Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning , pages 5530-5540. PMLR, 2021.                                                                                                                   |
Bozhou Li Hao Liang Zimo Meng Wentao Zhang
Are Bigger Encoders Always Better in Vision Large Models?
In recent years, multimodal large language models (MLLMs) have shown strong
potential in real-world applications. They are developing rapidly due to their
remarkable ability to comprehend multimodal information and their inherent
powerful cognitive and reasoning capabilities. Among MLLMs, vision language
models (VLM) stand out for their ability to understand vision information.
However, the scaling trend of VLMs under the current mainstream paradigm has
not been extensively studied. Whether we can achieve better performance by
training even larger models is still unclear. To address this issue, we
conducted experiments on the pretraining stage of MLLMs. We conduct our
experiment using different encoder sizes and large language model (LLM) sizes.
Our findings indicate that merely increasing the size of encoders does not
necessarily enhance the performance of VLMs. Moreover, we analyzed the effects
of LLM backbone parameter size and data quality on the pretraining outcomes.
Additionally, we explored the differences in scaling laws between LLMs and
VLMs.