HyperCLOVAX-Seed-Vision-3B

LICENSE

@@ -0,0 +1,62 @@
+HyperCLOVA X SEED Model License Agreement
+
+Model Release Date: April 24, 2025
+ 
+This HyperCLOVA X SEED Model License Agreement (the “Agreement”) is a legal agreement between you and NAVER Corporation and NAVER Cloud Corporation (“NAVER”) and governs your use of the Models that NAVER provides to You under this Agreement. 
+
+NAVER Corp., as the holder of the intellectual property of the Model, and its affiliate, NAVER Cloud Corp., as the exclusive business operator of HyperCLOVA X, enter into this Agreement with you. NAVER and you are each a “party” and collectively the “parties.”
+
+By using, reproducing, modifying, distributing, performing or displaying any portion or element of the Model or Derivative Model, or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement. You represent to us that you are lawfully able to enter into contracts, and if you are entering into this Agreement for an entity, that you have legal authority to bind that entity. 
+
+1.	Definitions. 
+
+1.1.	"Affiliate” means any entity directly or indirectly controlling, controlled by or under common control with either party, where “control” means the possession, directly or indirectly, of the power to independently direct or cause the direction of the management and policies of an entity, whether through ownership of more than fifty percent (50%) of the stock or other equity interests entitled to vote for representation on its board of directors, or body performing similar functions, by contract or otherwise.
+
+1.2.	“Derivative Model” means all (i) modifications to the Model, (ii) works based on the Model, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of the Model, to that model in order to cause that model to perform similarly to the Model, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by the Model for training that Model. For clarity, Outputs are not deemed Derivative Model. 
+
+1.3.	“Licensee” or “you” means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity’s behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
+
+1.4.	“Model” means the foundational large language models and software and algorithms, including machine-learning model code and trained model weights distributed by NAVER. 
+
+1.5.	“Output” means the information content output of the Model or a Derivative Model that results from operating or otherwise using the Model or Derivative Models. 
+
+2.	Conditions for Use, License Grant and Restrictions
+
+2.1.	Conditions for Use. The Model and any Derivative Model are subject to the terms of this Agreement and govern your use. If You institute copyright or patent litigation against any entity (including a crossclaim or counterclaim in a lawsuit) alleging that the Model or a Derivative Model constitutes direct or contributory copyright or patent infringement, then any license granted to you under this Agreement for that Model or Derivative Model will terminate as of the date such litigation is filed. NAVER may update this Agreement to comply with legal and regulatory requirements any time and You agree to either comply with any updated license or cease your copying, use, and distribution of the Model and any Derivative Model.
+
+2.2.	License Grant. Subject to the terms and conditions of this Agreement, NAVER hereby grants to you a non-exclusive, worldwide, non-transferable, revocable and royalty-free limited license under NAVER’s intellectual property or other rights owned by NAVER embodied in the Model to access, download, install, copy, use, reproduce, distribute, create derivative works of, and make modifications to the Model. 
+
+2.3.	Prohibited Use Policy. NAVER is committed to safety, trust and transparency in AI development. NAVER encourages You to (i) ensure that the product or service you develop, use, offer as a service or distributes meets the legal and ethical requirements of the relevant industry or use case, (ii) take reasonable measures to address unintended bias and to mitigate harm to others, including underrepresented or vulnerable groups, and (iii) inform users of the nature and limitations of the product or service. NAVER expressly prohibits the use of its products or services for any purpose in violation of applicable law and regulation, including but not limited to (a) illegal surveillance, (b) illegal collection or processing of biometric information without the consent of the subject where required under applicable law, or (c) illegal harassment, abuse, threatening or bullying of individuals or groups of individuals or intentionally misleading or deceiving others. 
+
+3.	Redistribution. 
+
+3.1.	You may reproduce, distribute or make available the Model or Derivative Models thereof, or a product or service (including another AI model) that contains any of them, if you meet all of the following conditions: you must (i) include the Prohibited Use Policy referenced in Section 2.3. as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of the Model or Derivative Model and you must provide notice to subsequence users you distribute to the Model or Derivative Models are subject to the use restrictions in Section 2.3., (ii) provide all third party recipients of the Model or Derivative Models a copy of this Agreement, (iii) cause any modified files to carry prominent notices stating that you modified the files; (iv) include the following attribution notice within a “Notice” text file distributed as part of such copies: “HyperCLOVA X SEED Model is licensed under the HyperCLOVA X SEED Model License Agreement, Copyright © NAVER Corp. All Rights Reserved.”, and (v) prominently display “Powered by HyperCLOVA X” on a related website, user interface, blogpost, about page, or product documentation. If you use the Model or any Outputs of the Model to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “HyperCLOVA X” at the beginning of any such AI model name.
+3.2.	You may add your own copyright statement to your modifications and, except as set forth in this Section, may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such Derivative Models as a whole, provided your use, reproduction, and distribution of the Model or Derivative Models otherwise comply with the terms and conditions stated in this Agreement. Any additional or different terms and conditions you impose must not conflict with the terms of this Agreement. 
+
+4.	Additional Commercial Terms. If (i) as of the Model Release Date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s Affiliates, is greater than 10 million monthly active users in the preceding calendar month, or (ii) the Licensee or its Affiliate distributes or makes available any product or service, which is substantially similar to or directly competes with any product and service provided by NAVER, then the Licensee must request a license from NAVER. Such license may be granted by NAVER at its sole discretion, and the Licensee is not authorized to exercise any rights under this Agreement unless and until NAVER expressly grants you such rights.
+
+5.	Generated Output. NAVER claims no rights in Outputs you generate using the Model. You and your use are solely responsible for Outputs and their subsequent uses. 
+
+6.	DISCLAIMER OF WARRANTY. UNLESS REQUIRED BY APPLICABLE LAW, THE MODEL AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR ANY KIND, AND NAVER DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MODEL, DERIVATIVE MODELS, OUTPUTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE MODEL AND ANY OUTPUTS AND RESULTS AND YOUR EXERCISE OF PERMISSION UNDER THIS AGREEMENT. 
+
+7.	LIMITATION OF LIABILITY. IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW (SUCH AS IN CASES OF DELIBERATE AND GROSSLY NEGLIGENT ACTS), WILL NAVER BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND, ARISING FROM OR RELATED TO THIS AGREEMENT, OR RESULTING FROMTHE USE OR INABILITY TO USE THE MODEL, DERIVATIVE MODELS OR, OUTPUTS (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGES, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES), EVEN IF NAVER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 
+
+8.	Indemnity. You will indemnify and hold harmless NAVER from and against any claim by any third party arising out of or related to your use or distribution of the Model, Derivative Model or Outputs. 
+
+9.	Intellectual Property.
+
+9.1.	This Agreement does not grant permission to use the trade names, trademarks, service marks, or product names of NAVER, except as required for reasonable and customary use in describing the origin of the Model and reproducing the content of the “Notice” text file. 
+
+9.2.	NAVER Corp. owns the Model and any Derivative Model created by NAVER Corp. Except as expressively granted in this Agreement, NAVER Corp. reserves all rights, interests and remedies in connection with the Model and Derivative Model created by NAVER Corp. and no other license or right is granted to you by implication, estoppel or otherwise. Subject to NAVER Corp.’s ownership of the Model and any Derivative Model made by or for NAVER Corp., with respect to any derivative works and modifications of the Model that are made by you, as between you and NAVER Corp., you are and will be the owner of such derivative works and modifications.
+
+10.	Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Model and will continue in full force and effect until terminated in accordance with the terms and conditions of this Agreement. NAVER may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Model and Derivative Model. Section 5, 6, 7 and 10 shall survive the termination of this Agreement. 
+
+11.	Governing Law and Jurisdiction. 
+
+11.1.	This Agreement will be governed by and construed in accordance with the laws of the Republic of Korea, without regard to its conflicts of laws principles. 
+
+11.2.	Any disputes, controversies, or claims arising out of or relating to this Agreement, including its existence, validity, interpretation, performance, breach, or termination, shall be referred to and finally resolved by arbitration administered by the Korean Commercial Arbitration Board (KCAB) in accordance with the International Arbitration Rules of the Korean Commercial Arbitration Board in force at the time of the commencement of the arbitration. The seat of arbitration shall be Seoul, Republic of Korea. The tribunal shall consist of one arbitrator. The language of the arbitration shall be English. Either party may seek interim or provisional relief from a court of competent jurisdiction, and doing so shall not be considered a waiver of any provision in this section. The arbitral tribunal also has the authority to issue orders for interim or provisional relief.
+
+12.	Modifications. NAVER reserves the right to modify or amend this Agreement at any time, in its sole discretion. Any modifications will be effective upon posting the updated Agreement on our website or through other means of communication. You are responsible for reviewing the Agreement periodically for changes. 
+
+13.	No Waiver. NAVER will not be treated as having waived any rights by not exercising (or delaying the exercise of) any rights under this Agreement.

__init__.py

@@ -0,0 +1,7 @@
+from transformers import AutoConfig, AutoModel, AutoModelForCausalLM
+
+from .configuration_hyperclovax import HCXVisionConfig
+from .modeling_hyperclovax import HCXVisionForCausalLM
+
+AutoConfig.register("hyperclovax_vlm", HCXVisionConfig)
+AutoModelForCausalLM.register(HCXVisionConfig, HCXVisionForCausalLM)

config.json

@@ -0,0 +1,190 @@
+{
+  "anyres": true,
+  "architectures": [
+    "HCXVisionForCausalLM"
+  ],
+  "decoder_max_length": 16384,
+  "freeze_decoder": false,
+  "freeze_encoder": true,
+  "freeze_mm_projector": false,
+  "hidden_size": 3072,
+  "ignore_index": -100,
+  "img_start_id": 100271,
+  "language_config": {
+    "add_cross_attention": false,
+    "architectures": [
+      "LlamaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 100257,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "end_token_id": 100257,
+    "eos_token_id": 100257,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 3072,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 7168,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "logits_scaling": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 131072,
+    "min_length": 0,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 24,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 100257,
+    "prefix": null,
+    "pretraining_tp": 1,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "resid_pdrop": 0.2,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 100000000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "transformers_version": "4.45.0",
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 110592
+  },
+  "max_num_grids": 9,
+  "model_type": "hyperclovax_vlm",
+  "max_image_cnt": 12,
+  "num_queries_vis_abstractor": 81,
+  "proj_pos_emb": true,
+  "proj_prenorm": false,
+  "q_former_model_name_or_path": null,
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.0",
+  "unpad": true,
+  "use_1x1_grid": true,
+  "use_nth_layer": -2,
+  "vision_config": {
+    "add_cross_attention": false,
+    "anyres": true,
+    "architectures": [
+      "SiglipVisionModel"
+    ],
+    "attention_dropout": 0.0,
+    "auto_map": {},
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 378,
+    "initializer_factor": 1.0,
+    "intermediate_size": 4304,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_num_grids": 9,
+    "min_length": 0,
+    "model_type": "siglip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 14,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "transformers_version": "4.45.0",
+    "typical_p": 1.0,
+    "use_bfloat16": true
+  }
+}

configuration_hyperclovax.py

@@ -0,0 +1,60 @@
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class HCXVisionConfig(PretrainedConfig):
+    model_type = "hyperclovax_vlm"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # The `gpt2` class has a different name, so it needs to be updated accordingly.
+    language_config_attribute_map = {
+        "n_embd": "hidden_size",
+        "n_positions": "max_position_embeddings",
+        "n_head": "num_attention_heads",
+        "n_layer": "num_hidden_layers",
+    }
+
+    def __init__(
+        self,
+        language_config=None,
+        vision_config=None,
+        use_nth_layer=-2,
+        img_start_id=100009,  # <|dummy3|>
+        decoder_max_length=4096,
+        anyres=False,
+        unpad=False,
+        max_num_grids=-1,
+        num_queries_vis_abstractor=-1,
+        ignore_index=-100,
+        proj_pos_emb=True,
+        proj_prenorm=False,
+        use_1x1_grid=False,
+        **kwargs,
+    ):
+        for key, val in self.language_config_attribute_map.items():
+            if language_config is not None and key in language_config:
+                language_config[val] = language_config.pop(key)
+
+        self.language_config = language_config
+        self.vision_config = vision_config
+
+        if language_config is not None:
+            # In DeepSpeed ZeRO-3, the memory size is automatically determined based on the `hidden_size` specified in the config.
+            self.hidden_size = (
+                language_config["hidden_size"] if "hidden_size" in language_config else language_config["n_embd"]
+            )
+        # add VLM configs
+        self.use_nth_layer = use_nth_layer
+        self.decoder_max_length = decoder_max_length
+        self.anyres = anyres
+        self.unpad = unpad
+        self.max_num_grids = max_num_grids
+        self.num_queries_vis_abstractor = num_queries_vis_abstractor
+        self.img_start_id = img_start_id
+        self.ignore_index = ignore_index
+        self.proj_pos_emb = proj_pos_emb
+        self.proj_prenorm = proj_prenorm
+        self.use_1x1_grid = use_1x1_grid
+        super().__init__(**kwargs)

modeling_hyperclovax.py

@@ -0,0 +1,1810 @@
+import ast
+import contextlib
+import gc
+import json
+import math
+import os
+from dataclasses import dataclass
+from functools import partial
+from itertools import chain
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+from einops import rearrange
+from timm.layers import LayerNorm, LayerNorm2d
+from timm.models.regnet import RegStage
+from torch.nn import CrossEntropyLoss
+from transformers import (
+    AutoConfig,
+    AutoModel,
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    PreTrainedModel,
+)
+from transformers.generation.utils import GenerationMixin
+from transformers.integrations.deepspeed import is_deepspeed_zero3_enabled
+from transformers.modeling_utils import (
+    is_fsdp_enabled,
+    is_local_dist_rank_0,
+    no_init_weights,
+)
+from transformers.models.auto import CONFIG_MAPPING
+from transformers.utils import ModelOutput
+
+from .configuration_hyperclovax import HCXVisionConfig
+from .preprocessor import select_best_resolution
+
+EOT = "<|endofturn|>"
+IMG_LOC = "<|dummy3|>"
+
+
+def get_rank():
+    if dist.is_initialized():
+        return dist.get_rank()
+    return 0
+
+
+def get_world_size():
+    if torch.distributed.is_initialized():
+        world_size = torch.distributed.get_world_size()
+    else:
+        world_size = 1
+    return world_size
+
+
+def unpad_image(tensor: torch.Tensor, original_size: Tuple[int, int]) -> torch.Tensor:
+    """Unpads a PyTorch tensor of a padded and resized image.
+
+    This function removes padding from a tensor image that was previously padded and resized.
+    The padding is removed based on the aspect ratio difference between the original and current image dimensions.
+
+    Args:
+        tensor: The image tensor, assumed to be in CxHxW format.
+        original_size: The original size of the image as (width, height).
+
+    Returns:
+        The unpadded image tensor.
+
+    Examples:
+        >>> import torch
+        >>> # Example 1: Unpadding with height padding
+        >>> padded_tensor = torch.randn(1, 64, 48)  # Padded tensor (C=1, H=64, W=48)
+        >>> original_size = (32, 32)  # Original size (width=32, height=32)
+        >>> unpadded_tensor = unpad_image(padded_tensor, original_size)
+        >>> unpadded_tensor.shape
+        torch.Size([1, 48, 48])
+        >>> # Example 2: Unpadding with width padding
+        >>> padded_tensor = torch.randn(1, 48, 64)  # Padded tensor (C=1, H=48, W=64)
+        >>> original_size = (32, 32)  # Original size (width=32, height=32)
+        >>> unpadded_tensor = unpad_image(padded_tensor, original_size)
+        >>> unpadded_tensor.shape
+        torch.Size([1, 48, 48])
+    """
+    original_width, original_height = original_size
+    current_height, current_width = tensor.shape[1:]
+
+    original_aspect_ratio = original_width / original_height
+    current_aspect_ratio = current_width / current_height
+
+    if original_aspect_ratio > current_aspect_ratio:
+        scale_factor = current_width / original_width
+        new_height = int(original_height * scale_factor)
+        padding = (current_height - new_height) // 2
+        unpadded_tensor = tensor[:, padding : current_height - padding, :]
+    else:
+        scale_factor = current_height / original_height
+        new_width = int(original_width * scale_factor)
+        padding = (current_width - new_width) // 2
+        unpadded_tensor = tensor[:, :, padding : current_width - padding]
+
+    return unpadded_tensor
+
+
+def get_anyres_image_grid_shape(
+    image_size: Tuple[int, int],
+    grid_pinpoints: Union[str, List[Tuple[int, int]]],
+    patch_size: int,
+) -> Tuple[int, int]:
+    """Calculates the image patch grid shape after any-resolution preprocessing.
+
+    Selects the optimal resolution from predefined grid pinpoints based on input image
+    dimensions using `select_best_resolution`, then computes the grid layout by
+    dividing the selected resolution by the patch size using integer division.
+
+    Args:
+        image_size (Tuple[int, int]): Original image dimensions in (width, height) format.
+        grid_pinpoints (Union[str, List[Tuple[int, int]]]): Accepts either:
+            - List of (height, width) resolution tuples
+            - String representation of list (e.g., "[(224, 224), (336, 336)]")
+        patch_size (int): Spatial dimension of square patches for grid division.
+
+    Returns:
+        Tuple[int, int]: Grid dimensions as (num_patches_width, num_patches_height).
+
+    Examples:
+        >>> # Basic case with list input
+        >>> get_anyres_image_grid_shape((1000, 800), [(224, 224), (448, 448)], 112)
+        (4, 4)
+
+        >>> # Basic case with string input
+        >>> get_anyres_image_grid_shape((600, 400), "[(336, 336), (672, 672)]", 112)
+        (6, 6)
+
+        >>> # Case where resolution is not perfectly divisible by patch_size
+        >>> # select_best_resolution picks (224, 224). 224 // 100 = 2
+        >>> get_anyres_image_grid_shape((500, 500), [(224, 224)], 100)
+        (2, 2)
+
+        >>> # Different patch size
+        >>> # select_best_resolution picks (448, 448). 448 // 224 = 2
+        >>> get_anyres_image_grid_shape((1200, 900), [(448, 448), (224, 224)], 224)
+        (2, 2)
+
+    Note:
+        String-formatted grid_pinpoints are converted via ast.literal_eval. Invalid formats
+        may raise syntax exceptions. The actual resolution selection depends on the
+        implementation of `select_best_resolution`. The doctests assume
+        `select_best_resolution` picks the *first* resolution provided in `grid_pinpoints`.
+    """
+    possible_resolutions = grid_pinpoints if isinstance(grid_pinpoints, list) else ast.literal_eval(grid_pinpoints)
+
+    original_width, original_height = image_size
+    height, width = select_best_resolution((original_height, original_width), possible_resolutions)
+    return width // patch_size, height // patch_size
+
+
+def reshape_and_unpad_image_features(
+    image_feature: torch.Tensor,
+    height: int,
+    width: int,
+    image_size: Tuple[int, int],
+    possible_resolutions: List[Tuple[int, int]],
+    grid_size: int,
+    unpad: bool,
+    image_newline: torch.Tensor,
+) -> torch.Tensor:
+    """Reshapes and processes image features with optional unpadding operation.
+
+    Processes input image features by:
+    1. Separating base features from spatial features
+    2. Reshaping spatial features into a 5D tensor (num_patch_height, num_patch_width, height, width, channels)
+    3. Performing either unpadding operation or simple reshaping based on 'unpad' flag
+    4. Concatenating processed features with base features
+
+    Args:
+        image_feature: Input tensor containing image features with shape
+            [1 + num_patches, feature_dim] where the first element is the base feature
+        height: Original image height in pixels
+        width: Original image width in pixels
+        image_size: Target image size as (width, height) tuple
+        possible_resolutions: List of possible [height, width] resolutions for multi-scale processing
+        grid_size: Grid dimension for patch arrangement
+        unpad: Flag to enable unpadding operation
+        image_newline: Special token tensor used as separator when unpadding
+
+    Returns:
+        torch.Tensor: Processed image features tensor with shape [1 + num_processed_patches, feature_dim]
+
+    Raises:
+        AssertionError: If base feature dimension doesn't match height*width
+    """
+    base_image_feature = image_feature[0]
+    image_feature = image_feature[1:]
+
+    assert (
+        height * width == base_image_feature.shape[0]
+    ), f"height: {height}, width: {width}, base_image_feature.shape[0]: {base_image_feature.shape[0]}"
+
+    num_patch_width, num_patch_height = get_anyres_image_grid_shape(image_size, possible_resolutions, grid_size)
+    image_feature = image_feature.view(num_patch_height, num_patch_width, height, width, -1)
+
+    if unpad:
+        image_feature = image_feature.permute(4, 0, 2, 1, 3).contiguous()
+        image_feature = image_feature.flatten(1, 2).flatten(2, 3)
+        image_feature = unpad_image(image_feature, image_size)
+        image_feature = torch.cat(
+            (
+                image_feature,
+                image_newline[:, None, None].expand(*image_feature.shape[:-1], 1).to(image_feature.device),
+            ),
+            dim=-1,
+        )
+        image_feature = image_feature.flatten(1, 2).transpose(0, 1)
+    else:
+        image_feature = image_feature.permute(0, 2, 1, 3, 4).contiguous()
+        image_feature = image_feature.flatten(0, 3)
+    image_feature = torch.cat((base_image_feature, image_feature), dim=0)
+
+    return image_feature
+
+
+def anyres_postprocessing(
+    image_forward_outs: torch.FloatTensor,
+    split_sizes: List[int],
+    image_sizes: List[List[int]],
+    possible_resolutions: List[Tuple[int, int]],
+    is_videos: List[bool],
+    patch_size: int,
+    grid_size: int,
+    image_newline: torch.FloatTensor,
+    num_queries_vis_abstractor: int = -1,
+    unpad: bool = False,
+) -> List[torch.FloatTensor]:
+    """Processes 2D visual features into 1D sequences with post-processing steps.
+
+    Performs AnyRes postprocessing by flattening 2D visual features from grid partitions into 1D sequences, adding
+    newline embeddings at row boundaries for images, and optionally removing padding regions based on original image
+    sizes. For video data, processes each frame's features separately into a single sequence per video and disables
+    unpadding and newline insertion.
+
+    Args:
+        image_forward_outs (List[torch.FloatTensor]): List of input tensors with shape
+            (number_of_images_in_grid, total_patches, feature_dim) containing visual features.
+        split_sizes (List[int]): A list containing the number of patches for each sample in the batch. The sum of
+            `split_sizes` should equal `image_forward_outs.shape[0]`.
+        image_sizes (List[List[int]]): A list where each element is a list `[width, height]` representing the original
+            dimensions of the corresponding image sample. Used for unpadding.
+        possible_resolutions (List[Tuple[int, int]]): A list of supported resolution tuples `(height, width)` used by
+            `reshape_and_unpad_image_features` for spatial reconstruction, especially during unpadding.
+        is_videos (List[bool]): A list of boolean flags indicating whether each corresponding sample in the batch is a
+            video [`True`] or an image [`False`].
+        patch_size (int): The spatial dimension (height and width) of the square patches the image was divided into.
+        grid_size (int): The spatial dimension (height and width) of the square grid onto which patches are mapped.
+            `grid_size` should be divisible by `patch_size`.
+        image_newline (torch.FloatTensor): A learnable tensor representing the newline embedding, typically with shape
+            (1, feature_dim). Added after each row of image patches when not unpadding.
+        num_queries_vis_abstractor (int, optional): If a visual abstractor with a fixed number of output queries is used
+            instead of grid patching, this specifies the number of queries. Must be a perfect square if > 0.
+            Defaults to -1 (indicating standard grid patching is used).
+        unpad (bool, optional): If `True`, removes padding tokens from image features based on `image_sizes` and
+            `possible_resolutions`. Does not apply to video features. Defaults to False.
+
+    Returns:
+        List[torch.FloatTensor]: A list of tensors, where each tensor represents the processed 1D sequence of visual
+            features for a single sample from the input batch. The length of the sequence varies depending on processing
+            (unpadding, newlines, video flattening).
+
+    Raises:
+        AssertionError: If `num_queries_vis_abstractor` is greater than 0 but not a perfect square.
+    """
+    height = width = grid_size // patch_size
+
+    if num_queries_vis_abstractor > 0:
+        assert (num_queries_vis_abstractor**0.5).is_integer(), "n_queries must be square number"
+        height = width = int(num_queries_vis_abstractor**0.5)
+
+    image_features = torch.split(image_forward_outs, split_sizes, dim=0)
+
+    # post-processing (unpad, add newline)
+    new_image_features = []
+    for image_idx, (image_feature, is_video) in enumerate(zip(image_features, is_videos)):
+        if image_feature.shape[0] > 1:
+            if not is_video:
+                image_feature = reshape_and_unpad_image_features(
+                    image_feature=image_feature,
+                    height=height,
+                    width=width,
+                    image_size=image_sizes[image_idx],
+                    possible_resolutions=possible_resolutions,
+                    grid_size=grid_size,  # Pass grid info if needed by helper
+                    unpad=unpad,
+                    image_newline=image_newline,
+                )
+            else:
+                image_feature = image_feature.flatten(0, 1)
+        else:
+            image_feature = image_feature[0]
+            if unpad and not is_video:
+                image_feature = torch.cat((image_feature, image_newline[None].to(image_feature.device)), dim=0)
+        new_image_features.append(image_feature)
+    image_features = new_image_features
+    return image_features
+
+
+def adaptive_anyres_postprocessing(
+    image_forward_outs: torch.FloatTensor,
+    image_sizes: List[List[int]],
+    possible_resolutions: List[Tuple[int, int]],
+    is_videos: List[bool],
+    group_ids: List[List[int]],
+    num_queries_vis_abstractors: List[List[int]],
+    grid_size: int,
+    image_newline: torch.FloatTensor,
+    unpad: bool = False,
+) -> List[torch.FloatTensor]:
+    """Adaptive AnyRes postprocessing for multi-group feature aggregation.
+
+    Processes 2D visual features into 1D sequences with group-wise adaptive processing. Each image can belong to
+    multiple processing groups with different query configurations. Features are processed per group and aggregated
+    according to group_ids.
+
+    Args:
+        image_forward_outs (List[torch.FloatTensor]): List of input tensors with shape
+            (number_of_images_in_grid, total_patches, feature_dim) containing visual features.
+        image_sizes (List[List[int]]): Original image dimensions for each sample. [[width, height], ... ]
+        possible_resolutions (List[Tuple[int, int]]): Supported resolutions. [[height, width], ... ]
+        is_videos (List[bool]): Flags indicating video inputs
+        group_ids (List[List[int]]): Group indices for feature aggregation. Each group means a single grid.
+        num_queries_vis_abstractors (List[List[int]]): Query numbers per group
+        grid_size (int): Total grid size for spatial processing
+        image_newline (torch.FloatTensor): Sample-wise config. Newline embedding tensor
+        unpad (bool, optional): Sample-wise config. Enable padding removal. Defaults to False.
+
+    Returns:
+        List[torch.FloatTensor]: Aggregated features per group
+
+    Raises:
+        AssertionError: If num_queries is not square number in any group
+    """
+    # post-processing (unpad, add newline)
+    new_image_features = []
+    for image_idx, (image_feature, is_video) in enumerate(zip(image_forward_outs, is_videos)):
+        num_queries_vis_abstractor = num_queries_vis_abstractors[image_idx]
+        assert (num_queries_vis_abstractor**0.5).is_integer(), "n_queries must be square number"
+        height = width = int(num_queries_vis_abstractor**0.5)
+
+        if image_feature.shape[0] > 1:
+            if not is_video:
+                image_feature = reshape_and_unpad_image_features(
+                    image_feature=image_feature,
+                    height=height,
+                    width=width,
+                    image_size=image_sizes[image_idx],
+                    possible_resolutions=possible_resolutions,
+                    grid_size=grid_size,
+                    unpad=unpad,
+                    image_newline=image_newline,
+                )
+            else:
+                image_feature = image_feature.flatten(0, 1)
+        else:
+            image_feature = image_feature[0]
+            if unpad and not is_video:
+                image_feature = torch.cat((image_feature, image_newline[None].to(image_feature.device)), dim=0)
+        new_image_features.append(image_feature)
+
+    image_features = [
+        torch.cat([new_image_features[group_id] for group_id in group_ids_list], dim=0) for group_ids_list in group_ids
+    ]
+    return image_features
+
+
+@dataclass
+class HCXVisionOutput(ModelOutput):
+    """Output class for vision models, containing various computation results.
+
+    Args:
+        loss (Optional[torch.FloatTensor], optional): Total cross-entropy loss calculated from logits and labels.
+        loss_per_sample (Optional[torch.FloatTensor], optional): Per-sample loss values for advanced loss processing.
+        logits (torch.FloatTensor): Classification scores (before SoftMax) of shape (batch_size, num_classes).
+        past_key_values (Optional[Tuple[Tuple[torch.FloatTensor]]], optional): Contains precomputed hidden-states
+            that can be used (see `past_key_values` input) to speed up sequential decoding.
+        hidden_states (Optional[Tuple[torch.FloatTensor]], optional):
+            Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
+            shape (batch_size, sequence_length, hidden_size).
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (Optional[Tuple[torch.FloatTensor]], optional): Tuple of torch.FloatTensor (one for each layer)
+            of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention
+            softmax, used to compute the weighted average in the self-attention heads.
+    """
+
+    loss: Optional[torch.FloatTensor] = None
+    loss_per_sample: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
+class HCXVisionForCausalLM(PreTrainedModel, GenerationMixin):
+    """HCX Vision model for causal language modeling with vision-language capabilities.
+
+    This class combines a vision model with a language model to create a multimodal model
+    capable of processing images or videos and generating text based on the visual inputs.
+
+    Attributes:
+        config_class: Configuration class for the model.
+        vision_model_name: Name of the vision model component.
+        _no_split_modules: List of modules that should not be split during parallel processing.
+        supports_gradient_checkpointing: Whether the model supports gradient checkpointing.
+        _skip_keys_device_placement: Keys to skip during device placement.
+    """
+
+    config_class = HCXVisionConfig
+    vision_model_name = "vision_model"
+    _no_split_modules = ["CLIPAttention", "SiglipVisionModel"]
+    supports_gradient_checkpointing = True
+    _skip_keys_device_placement = "past_key_values"
+
+    def __init__(
+        self,
+        config: HCXVisionConfig,
+        **kwargs: Optional[Any],
+    ) -> None:
+        """Initialize the HCXVisionForCausalLM model.
+
+        Args:
+            config: Configuration object for the model containing parameters for both
+                vision and language components.
+            **kwargs: Additional keyword arguments:
+                - use_liger: Whether to use liger kernel for hyperclovax models.
+                - use_fused_ce: Whether to use fused cross-entropy loss.
+                - use_sum_loss: Whether to use sum reduction for loss instead of mean.
+                - is_safetensor_save: Whether to save model using safetensors format.
+
+        Raises:
+            ValueError: If vision_config is not defined or if language_config is not defined.
+        """
+        super().__init__(config)
+
+        self.flag_changed_max_position_embeddings = False
+
+        vision_model_type = config.vision_config["model_type"]
+        if vision_model_type in CONFIG_MAPPING:
+            vision_config = CONFIG_MAPPING[vision_model_type](**config.vision_config)
+            vision_config.auto_map = {}
+        else:
+            if config.vision_model_name_or_path is not None:
+                vision_config = AutoConfig.from_pretrained(config.vision_model_name_or_path, trust_remote_code=True)
+            elif config.vision_config["_name_or_path"] is not None:
+                vision_config = AutoConfig.from_pretrained(
+                    config.vision_config["_name_or_path"], trust_remote_code=True
+                )
+            else:
+                raise ValueError("vision_config is not defined")
+
+        self.use_liger = kwargs.pop("use_liger", False)
+        self.use_fused_ce = kwargs.pop("use_fused_ce", False)
+        self.reduction = "sum" if kwargs.pop("use_sum_loss", False) else "mean"
+
+        self.vision_config = vision_config
+        vision_config.anyres = config.anyres
+        vision_config.max_num_grids = config.max_num_grids
+
+        possible_resolutions = []
+        if config.anyres:
+            assert config.max_num_grids > 0
+            for i in range(1, config.max_num_grids + 1):
+                for j in range(1, config.max_num_grids + 1):
+                    if i == 1 and j == 1 and not config.use_1x1_grid:
+                        continue
+                    if i * j <= config.max_num_grids:
+                        possible_resolutions.append([i, j])
+
+            possible_resolutions = [
+                [ys * vision_config.image_size, xs * vision_config.image_size] for ys, xs in possible_resolutions
+            ]
+
+        self.possible_resolutions = possible_resolutions
+
+        with no_init_weights():
+            self.vision_model = AutoModel.from_config(
+                vision_config, trust_remote_code=True
+            )  # weight will be loaded in from_pretrained
+
+        assert config.language_config["model_type"] == "llama"
+        language_config = CONFIG_MAPPING["llama"](**config.language_config)
+        language_config._attn_implementation = kwargs.get("attn_implementation", "sdpa")  # activate flash attention
+        language_config.logits_scaling = 1.0
+
+        self.language_config = language_config
+        self.language_model = AutoModelForCausalLM.from_config(language_config)
+
+        self.language_model.gradient_checkpointing_enable()
+        self.num_queries_vis_abstractor = config.num_queries_vis_abstractor
+
+        # mm_projctor(==connector); vision_model_hidden_size -> LLM embedding size
+        input_hidden_size = vision_config.hidden_size
+        self.mm_projector = HCXVisionCAbstractor(
+            num_queries=self.num_queries_vis_abstractor,
+            num_input_tokens=(self.vision_config.image_size // self.vision_config.patch_size) ** 2,
+            encoder_hidden_size=input_hidden_size,
+            hidden_size=input_hidden_size,
+            output_hidden_size=language_config.hidden_size,
+            pos_emb=config.proj_pos_emb,
+            prenorm=config.proj_prenorm,
+        )
+        self.use_nth_layer = config.use_nth_layer
+        self.config.update({"vision_config": self.vision_model.config.to_dict()})
+        self.config.update({"language_config": self.language_model.config.to_dict()})
+        self.lm_head_vocab_size = (
+            language_config.padded_vocab_size
+            if hasattr(language_config, "padded_vocab_size")
+            else language_config.vocab_size
+        )
+        self.language_model.lm_head = nn.Linear(language_config.hidden_size, self.lm_head_vocab_size, bias=False)
+        self.model_parallel = False
+        self.device_map = None
+        self.use_no_grad = None
+        self.decoder_max_length = config.decoder_max_length
+
+        self.anyres = config.anyres
+        self.unpad = config.unpad
+        if self.anyres:
+            self.image_newline = nn.Parameter(torch.empty(language_config.hidden_size, dtype=self.dtype))
+
+        self.is_safetensor_save = kwargs.get("is_safetensor_save", True)
+        self._backward_compatibility_gradient_checkpointing()
+
+    def _init_weights(self, module):
+        # copies from https://github.com/kakaobrain/honeybee/blob/main/honeybee/common_layers.py#L55
+        if (
+            isinstance(module, nn.Conv2d)  # noqa: SIM101
+            or isinstance(module, nn.Embedding)
+            or isinstance(module, nn.Linear)
+        ):
+            module.weight.data.normal_(mean=0.0, std=0.02)
+            if hasattr(module, "bias") and module.bias is not None:
+                module.bias.data.zero_()
+
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+        elif isinstance(module, nn.Parameter):
+            embed_std = 1 / torch.sqrt(torch.tensor(module.size(0), dtype=torch.float)).to(module.dtype)
+            module.data.normal_(mean=0.0, std=embed_std)
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[List[List[torch.FloatTensor]]] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        image_sizes: Optional[List[List[List[int]]]] = None,
+        vision_query_lengths: Optional[List[List[int]]] = None,
+        non_vision_query_lengths: Optional[List[int]] = None,
+        img_start_ids_list: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors_slow: Optional[List[List[int]]] = None,
+        first_last_frames_slows: Optional[List[bool]] = None,
+        is_video_list: Optional[List[bool]] = None,
+        **kwargs,
+    ) -> Union[Tuple, HCXVisionOutput]:
+        """Forward pass of the model.
+
+        This method processes the input tokens and images, combines them into a unified
+        representation, and generates text output based on the inputs.
+
+        Args:
+            input_ids: Input token IDs. In positions where images are inputted, the value is replaced by "<|dummy3|>"
+            pixel_values: List of lists of 4D tensors for images. Each outer list corresponds to a batch and contains
+                inner lists of image tensors.
+            past_key_values: Pre-computed key and value states of the attention layers for faster inference.
+            attention_mask: Mask to avoid performing attention on padding token indices.
+            inputs_embeds: Input embeddings. If provided, input_ids will not be used.
+            labels: Labels for computing the language modeling loss.
+            use_cache: Whether to use past key/values for faster inference.
+            output_attentions: Whether to return attention weights of each layer.
+            output_hidden_states: Whether to return hidden states of each layer.
+            return_dict: Whether to return a ModelOutput instead of a tuple.
+            image_sizes: List of lists representing image dimensions (width, height).
+            vision_query_lengths: List of lists containing lengths when each image is converted into visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            num_queries_vis_abstractors: List of lists containing number of visual tokens for each image grid.\
+                For video frames, this is the number of visual tokens for the fast part.
+            num_queries_vis_abstractors_slow: List of lists containing number of visual tokens for
+                the slow part when applying the slowfast algorithm to video frames.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is
+                applied to the first or last frames of the video.
+            is_video_list: List of booleans indicating which inputs are videos.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            If return_dict=True, returns an HCXVisionOutput object containing:
+                - loss: Language modeling loss if labels are provided, otherwise None.
+                - loss_per_sample: Per-sample loss if labels are provided, otherwise None.
+                - logits: Prediction scores of the language modeling head.
+                - past_key_values: Past key/values for faster inference if use_cache=True.
+                - hidden_states: Hidden states of all layers if output_hidden_states=True.
+                - attentions: Attention weights of all layers if output_attentions=True.
+            If return_dict=False, returns a tuple containing the above items except loss_per_sample.
+        """
+        output_attentions = (
+            output_attentions if output_attentions is not None else self.config.vision_config["output_attentions"]
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else self.config.vision_config["output_hidden_states"]
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if inputs_embeds is None and past_key_values is None:
+            inputs_embeds = self.extract_inputs_embeds(
+                input_ids=input_ids,
+                pixel_values=pixel_values,
+                past_key_values=past_key_values,
+                image_sizes=image_sizes,
+                vision_query_lengths=vision_query_lengths,
+                non_vision_query_lengths=non_vision_query_lengths,
+                img_start_ids_list=img_start_ids_list,
+                num_queries_vis_abstractors=num_queries_vis_abstractors,
+                num_queries_vis_abstractors_slow=num_queries_vis_abstractors_slow,
+                first_last_frames_slows=first_last_frames_slows,
+                is_videos=is_video_list,
+            )
+
+        if inputs_embeds is not None:
+            input_ids = None
+
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.language_model.base_model(
+            input_ids=input_ids,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        hidden_states = hidden_states * self.language_config.logits_scaling
+
+        loss = None
+        loss_per_sample = None
+        logits = self.language_model.lm_head(hidden_states)
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss(reduction="none")  # ignore IGNORE_INDEX(-100)
+            shift_logits = shift_logits.view(-1, self.lm_head_vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model/pipeline parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+            if get_rank() == 0:
+                loss_per_sample = loss.view(logits.shape[0], -1).sum(axis=1) / (
+                    shift_labels.view(logits.shape[0], -1) != self.config.ignore_index
+                ).sum(axis=1)
+            loss = loss[shift_labels != self.config.ignore_index].mean()
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return HCXVisionOutput(
+            loss=loss,
+            loss_per_sample=loss_per_sample,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def determine_non_vision_query_lengths(
+        self, input_ids: torch.LongTensor, pad_id: int, img_start_id: int
+    ) -> List[int]:
+        """Calculate the lengths of non-vision query parts in the input.
+
+        This method calculates the length of text tokens (excluding visual tokens) for each sample.
+        When input_ids are collated, they are padded with pad_id on the right, so this method finds
+        these values by identifying pad tokens and img_start_id tokens.
+
+        Args:
+            input_ids: Input token IDs with img_start_id markers for image positions.
+            pad_id: Token ID used for padding.
+            img_start_id: Token ID marking the start of image data.
+
+        Returns:
+            List of lengths of non-vision query parts for each sample in the batch.
+        """
+        non_vision_query_lengths = []
+        batch_size, len_seq = input_ids.size(0), input_ids.size(1)
+
+        for i in range(batch_size):
+            temp_idx = (input_ids[i] == pad_id).nonzero()
+            eos_idx = temp_idx[0, 0].item() if len(temp_idx) > 0 else len_seq
+            num_imgs = (input_ids[i] == img_start_id).sum().item()
+            non_vision_query_lengths.append(eos_idx - num_imgs)
+
+        if all([pad_id in input_id for input_id in input_ids.tolist()]):
+            non_vision_query_lengths = [
+                non_vision_query_length + 1 for non_vision_query_length in non_vision_query_lengths
+            ]
+
+        return non_vision_query_lengths
+
+    def determine_vision_query_lengths(
+        self, image_features: List[List[torch.Tensor]], image_cnts: List[int]
+    ) -> List[List[int]]:
+        """Calculate the lengths of vision query parts in the input.
+
+        This method calculates the lengths of visual tokens for each image in each sample based on
+        the shapes of image feature tensors. For samples without any images, a dummy image is included
+        but then converted to an empty list.
+
+        Args:
+            image_features: List of lists of image features tensors.
+            image_cnts: List of counts of images for each sample in the batch.
+
+        Returns:
+            List of lists of lengths of visual tokens for each image in each sample.
+        """
+        vision_query_lengths = [
+            [image_feature.size(0) for image_feature in image_feature_list] for image_feature_list in image_features
+        ]
+
+        for i, image_cnt in enumerate(image_cnts):
+            if image_cnt == 0:
+                assert len(vision_query_lengths[i]) == 1  # 현재 검정 이미지 1개 들어가있음
+                vision_query_lengths[i] = []  # 빈 list 로 변환
+
+        return vision_query_lengths
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_input_embeddings
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_input_embeddings
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_output_embeddings
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_output_embeddings
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_decoder
+    def set_decoder(self, decoder):
+        self.language_model.set_decoder(decoder)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_decoder
+    def get_decoder(self):
+        return self.language_model.get_decoder()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.tie_weights
+    def tie_weights(self):
+        return self.language_model.tie_weights()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.resize_token_embeddings
+    def resize_token_embeddings(self, new_num_tokens: Optional[int] = None, pad_to_multiple_of=None) -> nn.Embedding:
+        model_embeds = self.language_model.resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
+        self.config.text_config.vocab_size = model_embeds.num_embeddings
+        self.vocab_size = model_embeds.num_embeddings
+        return model_embeds
+
+    def extract_inputs_embeds(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[List[List[torch.FloatTensor]]] = None,  # list of list of 4D tensors
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        image_sizes: Optional[List[List[List[int]]]] = None,
+        vision_query_lengths: Optional[List[List[int]]] = None,
+        non_vision_query_lengths: Optional[List[int]] = None,
+        img_start_ids_list: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors_slow: Optional[List[List[int]]] = None,
+        first_last_frames_slows: Optional[List[bool]] = None,
+        is_videos: Optional[List[str]] = None,
+    ):
+        """Extract input embeddings by processing text tokens and visual features.
+
+        This method processes the input tokens and image features, extracts the visual features
+        using the vision model, and combines them with the text token embeddings to create
+        a unified input representation for the language model.
+
+        Args:
+            input_ids: Input token IDs with img_start_id markers for image positions.
+            pixel_values: List of lists of image tensors.
+            past_key_values: Pre-computed key and value states for faster inference.
+            image_sizes: List of lists of image dimensions (width, height).
+            vision_query_lengths: List of lists of lengths when each image is converted to visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            num_queries_vis_abstractors: List of lists containing number of visual tokens for each image grid.
+            num_queries_vis_abstractors_slow: List of lists containing number of visual tokens for
+                the slow part when applying the slowfast algorithm to video frames.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is
+                applied to the first or last frames of the video.
+            is_videos: List of booleans indicating which inputs are videos.
+
+        Returns:
+            Combined embeddings of text tokens and visual features.
+        """
+        inputs_embeds = None
+        if past_key_values:
+            pass
+        else:
+            # Flatten CLIP and connector for feature encoding, then convert back to List of List format
+            len_pixel_values = [len(pixel_value) for pixel_value in pixel_values]
+            concat_pixel_values = torch.cat(list(chain(*pixel_values)), dim=0)  # list of list of 4D Tensor
+            visual_token_idx = 0 if "siglip" in self.vision_config.model_type else 1
+            # Check if all parameters of the model require_grad=False
+            if self.use_no_grad is None:
+                self.use_no_grad = all(not p.requires_grad for p in self.vision_model.vision_model.encoder.parameters())
+            context = torch.no_grad() if self.use_no_grad else contextlib.nullcontext()
+            with context:
+                if self.use_no_grad:
+                    # Fixed number of for-loop iterations to 10.
+                    # Currently no memory effect observed, so proceeding without chunking.
+                    n_chunks = 1
+                else:
+                    n_chunks = 1
+                total_len = concat_pixel_values.size(0)
+                # Calculate the size of each chunk based on total data length (divided into 10 chunks)
+                chunk_size = math.ceil(total_len / n_chunks) if total_len > 0 else 1
+                image_forward_outs_chunks = []
+
+                for i in range(n_chunks):
+                    start = i * chunk_size
+                    end = (i + 1) * chunk_size
+                    # Current chunk slice (could be an empty tensor if there's no data)
+                    chunk = concat_pixel_values[start:end].to(self.vision_model.dtype)
+                    # If the current chunk size is smaller than chunk_size, pad with dummy data
+                    if chunk.size(0) < chunk_size:
+                        # print(f"chunk.size(0): {chunk.size(0)}, chunk_size: {chunk_size}")
+                        pad_size = chunk_size - chunk.size(0)
+                        # Create dummy tensor based on concat_pixel_values shape
+                        dummy_shape = (pad_size,) + tuple(concat_pixel_values.shape[1:])
+                        dummy = torch.zeros(
+                            dummy_shape,
+                            dtype=concat_pixel_values.dtype,
+                            device=concat_pixel_values.device,
+                        )
+                        chunk = torch.cat([chunk, dummy], dim=0)
+
+                    # Pass the chunk through the vision model (processed according to use_nth_layer)
+                    if self.use_nth_layer == -1:
+                        # Replace post_layernorm of the last layer with Identity
+                        self.vision_model.vision_model.post_layernorm = nn.Identity()
+                        outs = self.vision_model(chunk)
+                        outs = outs.last_hidden_state[:, visual_token_idx:]
+                    else:
+                        outs = self.vision_model(chunk, output_hidden_states=True)
+                        outs = outs.hidden_states[self.use_nth_layer][:, visual_token_idx:]
+                    image_forward_outs_chunks.append(outs)
+
+                # Concatenate results from all chunks
+                image_forward_outs = torch.cat(image_forward_outs_chunks, dim=0).to(image_forward_outs_chunks[0].dtype)
+
+            if num_queries_vis_abstractors is None:
+                assert num_queries_vis_abstractors_slow is None
+                image_sizes = list(chain(*image_sizes))
+                if is_videos is not None:
+                    is_videos = list(chain(*is_videos))
+                group_ids = None
+                image_forward_outs = image_forward_outs.to(dtype=self.mm_projector.dtype)
+                image_forward_outs = self.mm_projector(image_forward_outs)
+            else:
+                # adaptive anyres is only implemented in HCXVisionCAbstractor
+                assert isinstance(self.mm_projector, HCXVisionCAbstractor)
+
+                (
+                    num_queries_vis_abstractors,
+                    num_grids,
+                    image_sizes,
+                    is_videos,
+                    group_ids,
+                ) = self.compute_adaptive_params(
+                    pixel_values,
+                    num_queries_vis_abstractors,
+                    num_queries_vis_abstractors_slow,
+                    image_sizes,
+                    is_videos,
+                    first_last_frames_slows,
+                )
+
+                image_forward_outs = image_forward_outs.to(dtype=self.mm_projector.dtype)
+                image_forward_outs = self.mm_projector(
+                    image_forward_outs,
+                    num_queries_vis_abstractors=num_queries_vis_abstractors,
+                    num_grids=num_grids,
+                )
+
+            if self.anyres:
+                split_sizes = [pixel_value.shape[0] for pixel_value in chain(*pixel_values)]
+
+                if num_queries_vis_abstractors is None:
+                    image_features = anyres_postprocessing(
+                        image_forward_outs=image_forward_outs,
+                        split_sizes=split_sizes,
+                        image_sizes=image_sizes,
+                        num_queries_vis_abstractor=self.num_queries_vis_abstractor,
+                        unpad=self.unpad,
+                        is_videos=is_videos,
+                        patch_size=self.vision_model.config.patch_size,
+                        grid_size=self.vision_model.config.image_size,
+                        image_newline=self.image_newline,
+                        possible_resolutions=self.possible_resolutions,
+                    )
+                else:
+                    image_features = adaptive_anyres_postprocessing(
+                        image_forward_outs=image_forward_outs,
+                        image_sizes=image_sizes,
+                        num_queries_vis_abstractors=num_queries_vis_abstractors,
+                        unpad=self.unpad,
+                        is_videos=is_videos,
+                        grid_size=self.vision_model.config.image_size,
+                        image_newline=self.image_newline,
+                        possible_resolutions=self.possible_resolutions,
+                        group_ids=group_ids,
+                    )
+            else:
+                if num_queries_vis_abstractors is None:
+                    image_features = [image_forward_out for image_forward_out in image_forward_outs]
+                else:
+                    image_features = [image_forward_out.unsqueeze(0) for image_forward_out in image_forward_outs]
+
+            # print(f"BEFORE GROUPING: len(image_features): {len(image_features)}")
+            image_features = [
+                image_features[sum(len_pixel_values[:i]) : sum(len_pixel_values[: i + 1])]
+                for i in range(len(len_pixel_values))
+            ]
+
+            batch_size = input_ids.size(0)
+            image_feature_dim = image_features[0][0].size(1)
+            image_feature_dtype = image_features[0][0].dtype
+
+            if img_start_ids_list is None:
+                image_cnts = (input_ids == self.config.img_start_id).sum(dim=1).tolist()
+            else:
+                image_cnts = [len(img_start_ids) for img_start_ids in img_start_ids_list]
+
+            if non_vision_query_lengths is None:
+                non_vision_query_lengths = self.determine_non_vision_query_lengths(
+                    input_ids, self.tokenizer.pad_token_id, self.config.img_start_id
+                )
+
+            if vision_query_lengths is None:
+                vision_query_lengths = self.determine_vision_query_lengths(image_features, image_cnts)
+
+            # Slicing is faster than concatenation
+            len_inputs_embeds = max(
+                [
+                    sum(vision_query_length) + non_vision_query_length
+                    for non_vision_query_length, vision_query_length in zip(
+                        non_vision_query_lengths, vision_query_lengths
+                    )
+                ]
+            )
+            len_inputs_embeds = min(self.decoder_max_length, len_inputs_embeds)
+
+            inputs_embeds = torch.zeros(
+                [batch_size, len_inputs_embeds, image_feature_dim],
+                dtype=image_feature_dtype,
+                device=self.device,
+                requires_grad=True,
+            ).clone()
+            # temp_embeds : torch.bfloat16 : [batchsize, 174, 3072]
+            temp_embeds = self.get_input_embeddings()(input_ids)
+
+            # The complete format is <PROMPT><USER_PREFIX><VISION_QUERIES>Sentence
+            for batch_idx, sample in enumerate(input_ids):
+                # Concatenate with visual tokens and then slice
+                non_vision_query_length = non_vision_query_lengths[batch_idx]
+                # Safely concatenate with visual tokens and then slice
+                sample = sample[: non_vision_query_length + image_cnts[batch_idx]]
+
+                if image_cnts[batch_idx] == 0:  # Text instruction data doesn't insert image features
+                    temp_idx = 0
+                    # Reference: https://github.com/haotian-liu/LLaVA/commit/44e0562f9497fb79f042427307472a87d266d90a#diff-4477387d506ccb1897a13972cba26c9da3fad4d3e1c32ec4b8bd8ff7acd3f292
+                    # https://github.com/intel/intel-extension-for-transformers/issues/1201#issuecomment-1915875119
+                    inputs_embeds[batch_idx, :non_vision_query_length] = temp_embeds[batch_idx][
+                        :non_vision_query_length
+                    ]
+                    inputs_embeds[batch_idx, temp_idx:temp_idx] = image_features[batch_idx][0][
+                        0:0
+                    ]  # First image of batch_idx sample (dummy image)
+                else:
+                    if img_start_ids_list is None:
+                        img_start_ids = (sample == self.config.img_start_id).nonzero()
+                    else:
+                        img_start_ids = img_start_ids_list[batch_idx]
+                    assert len(img_start_ids) == image_cnts[batch_idx] == len(image_features[batch_idx])
+                    # Initialize starting points for input embeddings and temporary embeddings
+                    input_start, temp_start = 0, 0
+
+                    # Iterate through each image starting point in the batch
+                    for multi_img_idx, img_start_idx in enumerate(img_start_ids):
+                        # Calculate token length up to the current image starting point
+                        token_len = img_start_idx - temp_start
+
+                        # Copy tokens to inputs_embeds
+                        inputs_embeds[batch_idx, input_start : input_start + token_len] = temp_embeds[
+                            batch_idx, temp_start : temp_start + token_len
+                        ]
+
+                        inputs_embeds[
+                            batch_idx,
+                            input_start
+                            + token_len : input_start
+                            + token_len
+                            + vision_query_lengths[batch_idx][multi_img_idx],
+                        ] = image_features[batch_idx][multi_img_idx]
+
+                        # Update starting points for next token processing
+                        input_start += token_len + vision_query_lengths[batch_idx][multi_img_idx]
+                        temp_start += token_len + 1  # Increase by 1 to skip the image start token
+
+                    # Process tokens after the last image end token
+                    token_len = min(sample[temp_start:].size(0), inputs_embeds.size(1) - input_start)
+                    inputs_embeds[batch_idx, input_start : input_start + token_len] = temp_embeds[
+                        batch_idx, temp_start : temp_start + token_len
+                    ]
+        return inputs_embeds
+
+    @torch.no_grad()
+    def generate(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[List[List[torch.FloatTensor]]] = None,
+        image_sizes: Optional[List[List[List[int]]]] = None,
+        vision_query_lengths: Optional[List[List[int]]] = None,
+        non_vision_query_lengths: Optional[List[int]] = None,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors_slow: Optional[List[List[int]]] = None,
+        first_last_frames_slows: Optional[List[bool]] = None,
+        is_videos: Optional[List[bool]] = None,
+        img_start_ids_list: Optional[List[List[int]]] = None,
+        pad_token_id: Optional[int] = None,
+        eos_token_id: Optional[int] = None,
+        bad_words_ids: Optional[List[List[int]]] = None,
+        max_length: int = 196,
+        min_length: int = 2,
+        do_sample: bool = True,
+        num_beams: int = 1,
+        top_p: float = 0.6,
+        top_k: int = 0,
+        temperature: float = 0.5,
+        repetition_penalty: float = 1.0,
+        length_penalty: int = 1,
+        use_cache: bool = True,
+        **kwargs,
+    ) -> torch.LongTensor:
+        """Generate text based on input tokens and images.
+
+        This method generates text based on the provided input tokens and images using
+        beam search and/or sampling strategies.
+
+        Args:
+            input_ids: Input token IDs with img_start_id markers for image positions.
+            pixel_values: List of lists of image tensors.
+            image_sizes: List of lists of image dimensions (width, height).
+            vision_query_lengths: List of lists of lengths when each image is converted to visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            num_queries_vis_abstractors: List of lists containing number of visual tokens for each image grid.
+            num_queries_vis_abstractors_slow: List of lists containing number of visual tokens for the slow part when
+                applying the slowfast algorithm to video frames.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is applied to the first
+                or last frames of the video.
+            is_videos: List of booleans indicating which inputs are videos.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            pad_token_id: Token ID used for padding.
+            eos_token_id: Token ID used to signal the end of a sequence.
+            bad_words_ids: List of token ID sequences that should not be generated.
+            max_length: Maximum length of the sequence to be generated (input length + max_new_tokens).
+            min_length: Minimum length of the sequence to be generated (input length + min_new_tokens).
+            do_sample: Whether to use sampling for generation (otherwise uses greedy decoding).
+            num_beams: Number of beams for beam search. 1 means no beam search.
+            top_p: Nucleus sampling parameter. Tokens with cumulative probability > top_p are kept.
+            top_k: Number of highest probability tokens to keep for top-k-filtering.
+            temperature: Value used to modulate the next token probabilities.
+            repetition_penalty: Penalty applied to tokens that have already appeared in the sequence.
+            length_penalty: Exponential penalty applied to sequence length.
+            use_cache: Whether to use past key/values for faster inference.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            Generated token IDs.
+        """
+        # inputs_embeds: torch.bfloat16 : [batchsize, variable(visual token, text token, system prompt 모두 포함)]
+        if pad_token_id is None:
+            pad_token_id = self.tokenizer.pad_token_id
+        if eos_token_id is None:
+            eos_token_id = self.tokenizer.encode("<|endofturn|>")[0]
+        if bad_words_ids is None:
+            bad_words_ids = [
+                [
+                    self.config.language_config["bos_token_id"],
+                ],
+                [
+                    self.config.language_config["eos_token_id"],
+                ],
+            ]
+
+        if pixel_values is None:
+            return self.language_model.generate(
+                input_ids, pad_token_id=pad_token_id, eos_token_id=eos_token_id, bad_words_ids=bad_words_ids, **kwargs
+            )
+        inputs_embeds = self.extract_inputs_embeds(
+            input_ids=input_ids,
+            pixel_values=self.to_vision_model_device(pixel_values),
+            image_sizes=image_sizes,
+            vision_query_lengths=vision_query_lengths,
+            non_vision_query_lengths=non_vision_query_lengths,
+            img_start_ids_list=img_start_ids_list,
+            num_queries_vis_abstractors=num_queries_vis_abstractors,
+            num_queries_vis_abstractors_slow=num_queries_vis_abstractors_slow,
+            first_last_frames_slows=first_last_frames_slows,
+            is_videos=is_videos,
+        )
+        inputs_embeds = (
+            inputs_embeds.to(self.base_model.device) if isinstance(inputs_embeds, torch.Tensor) else inputs_embeds
+        )
+
+        # pred : torch.int64 : [batchsize, generated token_length]
+        pred = self.language_model.generate(
+            inputs_embeds=inputs_embeds,
+            pad_token_id=pad_token_id,
+            eos_token_id=eos_token_id,
+            bad_words_ids=bad_words_ids,
+            max_new_tokens=max_length,
+            min_length=min_length,
+            num_beams=num_beams,
+            do_sample=(False if temperature == 0.0 else do_sample),  # set do_sample=False if invalid temperature
+            top_k=top_k,
+            top_p=top_p,
+            temperature=temperature,
+            repetition_penalty=repetition_penalty,
+            length_penalty=length_penalty,
+            early_stopping=(False if num_beams <= 1 else True),  # set early_stopping=False when not beam_search
+            use_cache=use_cache,
+        )
+
+        return pred
+
+    def to_vision_model_device(self, input_tensor: Union[torch.Tensor, List]) -> Union[torch.Tensor, List]:
+        """Move input tensors to the vision model's device.
+        This method recursively moves input tensors or lists of tensors to the vision model's device.
+
+        Args:
+            input_tensor: Input tensor or list of tensors to be moved to the vision model's device.
+
+        Returns:
+            The input tensor or list of tensors moved to the vision model's device.
+
+        Raises:
+            TypeError: If the input is neither a tensor nor a list.
+        """
+        if isinstance(input_tensor, list):
+            return [self.to_vision_model_device(item) for item in input_tensor]
+        elif isinstance(input_tensor, torch.Tensor):
+            return input_tensor.to(self.vision_model.device)
+        else:
+            raise TypeError("Unsupported data type. Only tensors and lists are allowed.")
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids: torch.LongTensor,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        **kwargs,
+    ) -> Dict[str, Any]:
+        """Prepare inputs for the generation algorithm.
+
+        This method prepares the input for each generation step based on the model's needs.
+
+        Args:
+            input_ids: Input token IDs.
+            past_key_values: Pre-computed key and value states for faster inference.
+            attention_mask: Mask to avoid performing attention on padding token indices.
+            inputs_embeds: Input embeddings. If provided, input_ids will not be used.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            Dictionary containing the prepared inputs for the model.
+        """
+        input_ids = kwargs.get("decoder_input_ids", input_ids)
+
+        if past_key_values:
+            input_ids = input_ids[:, -1:]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "pixel_values": kwargs.get("pixel_values", None),
+            }
+        )
+        return model_inputs
+
+    @classmethod
+    def from_config(cls, config, vision_model_name_or_path):
+        return cls(config, vision_model_name_or_path)
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path: Optional[Union[str, os.PathLike]] = None,
+        *model_args,
+        **kwargs,
+    ) -> "HCXVisionForCausalLM":
+        assert pretrained_model_name_or_path is not None
+
+        save_only_vision = kwargs.pop("save_only_vision") if "save_only_vision" in kwargs else False
+        save_only_qformer = kwargs.pop("save_only_qformer") if "save_only_qformer" in kwargs else False
+        save_shard_size = kwargs.pop("save_shard_size") if "save_shard_size" in kwargs else "5GB"
+
+        if pretrained_model_name_or_path is not None:  # when evaluate or load instruction tunned model
+            model: HCXVisionForCausalLM = super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
+            model.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)
+
+        img_start_id = model.tokenizer.encode(IMG_LOC, add_special_tokens=False)
+        assert (
+            len(img_start_id) == 1
+        ), f'"<|dummy3|>" was not encoded into a single special token. Encoding result: {img_start_id}'
+        model.config.img_start_id = img_start_id[0]
+
+        model.save_only_vision = save_only_vision
+        model.save_only_qformer = save_only_qformer
+        model.save_shard_size = save_shard_size
+
+        return model
+
+    def get_language_model(self):
+        return self.language_model.base_model
+
+    def get_vision_model(self):
+        return self.vision_model
+
+    def save_pretrained(
+        self,
+        save_directory: Union[str, os.PathLike],
+        *args,
+        **kwargs,
+    ):
+        state_dict = kwargs["state_dict"] if "state_dict" in kwargs else self.state_dict()
+        partial_state_dict = self.get_pretrained_state_dict(
+            state_dict,
+            save_directory,
+        )
+        kwargs["state_dict"] = partial_state_dict
+        kwargs["safe_serialization"] = self.is_safetensor_save
+        kwargs.setdefault("max_shard_size", self.save_shard_size)
+        super().save_pretrained(save_directory, *args, **kwargs)
+
+    def get_pretrained_state_dict(self, state_dict, save_dir):
+        vision_key = "vision_model."
+        llm_keys = ["language_model."]
+        head_key = "lm_head."
+
+        for key in list(state_dict.keys()):
+            if self.save_only_vision:
+                for llm_key in llm_keys:
+                    if llm_key in key:
+                        state_dict.pop(key)
+                if key.startswith(head_key):
+                    state_dict.pop(key)
+
+                elif self.save_only_qformer:
+                    if f"{vision_key}" in key:
+                        state_dict.pop(key)
+
+        return state_dict
+
+    def compute_adaptive_params(
+        self,
+        pixel_values: Optional[List[List[torch.FloatTensor]]] = None,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_queries_vis_abstractors_slow: Optional[List[List[int]]] = None,
+        image_sizes: Optional[List[List[List[int]]]] = None,
+        is_videos: Optional[List[bool]] = None,
+        first_last_frames_slows: Optional[List[bool]] = None,
+    ) -> Tuple[List[int], List[int], List[List[int]], List[bool], List[List[int]]]:
+        """Compute adaptive parameters for processing different image and video inputs.
+
+        This method calculates parameters needed for adaptive processing, especially when handling
+        variable resolutions or applying the slowfast algorithm to video frames. It flattens
+        batch-level inputs (lists of lists) into single lists representing all images/frames
+        in the batch. Based on slowfast configuration, it may split video frames into 'slow'
+        and 'fast' components, adjusting query counts and grid indices accordingly.
+
+        Args:
+            pixel_values: List of lists of image tensors (per sample). Used to determine the initial number of grids per
+                image/frame.
+            num_queries_vis_abstractors: List of lists (per sample) containing the base number of visual tokens
+                generated by the visual abstractor for each image grid
+                (e.g., 81 for a full grid, 9 for a subsampled/fast grid).
+            num_queries_vis_abstractors_slow: List of lists (per sample) containing the number of visual tokens for the
+                'slow' path when applying slowfast. Non-zero values here trigger the slowfast processing logic.
+            image_sizes: List of lists (per sample) of original image dimensions ([width, height]).
+            is_videos: List of lists (per sample) of booleans indicating if each input item is part of a video sequence.
+            first_last_frames_slows: List (per sample) of booleans. If True, slowfast logic
+                (if active based on `num_queries_vis_abstractors_slow`) is applied only to the first or last frame(s)
+                within each video sequence.
+
+        Returns:
+            Tuple containing:
+                - num_queries_vis_abstractors: Flattened list of final query counts per processed grid.
+                  Values might be adjusted based on slow/fast splitting
+                  (e.g., using values from `num_queries_vis_abstractors_slow` for slow frames).
+                  Example: [81, 81, 81, 9, 81, 9, ...] (Image, Image, Vid_Slow, Vid_Fast, Vid_Slow, Vid_Fast...)
+                - num_grids: Flattened list representing cumulative grid counts, acting as end indices for slicing the
+                  flattened `image_forward_outs`. Adjusted for slow/fast splits.
+                  Example: [0, 1, 9, 10, 18, 19, 27, ...] (Indices after Grid0_Slow(1),
+                  Grid1_Fast(8), Grid2_Slow(1), Grid3_Fast(8)...).
+                - image_sizes: Flattened list of image dimensions ([width, height]), potentially duplicated if slow/fast
+                  splitting occurred.
+                - is_videos: Flattened list of booleans indicating video status, potentially duplicated for
+                  slow/fast splits. Example: [False, False, True, True, True, True, ...]
+                  (Image1, Image2, Vid_grid1_slow, Vid_grid1_fast, Vid_grid2_slow, Vid_grid2_fast...)
+                - group_ids: List of lists, grouping indices that correspond to the same original image or frame.
+                  If a frame is split into slow/fast, its group will contain multiple indices.
+                  Example: [[0], [1], [2, 3], [4, 5], ...]
+                  (Group for Image1, Group for Image2, Group for Vid1_Slow+Fast, Group for Vid2_Slow+Fast...).
+
+        Raises:
+            AssertionError: If input validation fails (e.g., negative query counts).
+            Exception: If an unexpected case is encountered during slowfast processing.
+        """
+
+        # Check if all elements are integers greater than or equal to 0
+        assert all(
+            all(isinstance(value, int) and value >= 0 for value in sublist) for sublist in num_queries_vis_abstractors
+        ), "All values in num_queries_vis_abstractors must be integers >= 0."
+
+        assert all(
+            all(isinstance(value, int) and value >= 0 for value in sublist)
+            for sublist in num_queries_vis_abstractors_slow
+        ), "All values in num_queries_vis_abstractors_slow must be integers >= 0."
+
+        assert is_videos is not None
+
+        # Is it the first or last image? (for applying slowfast to video processing)
+        is_first_images = []
+        is_last_images = []
+        for is_video in is_videos:
+            for idx, is_video_item in enumerate(is_video):
+                if idx == 0:
+                    is_first_images.append(True)
+                else:
+                    is_first_images.append(False)
+                if idx == len(is_video) - 1:
+                    is_last_images.append(True)
+                else:
+                    is_last_images.append(False)
+
+        num_queries_vis_abstractors = list(chain(*num_queries_vis_abstractors))
+        num_queries_vis_abstractors_slow = list(chain(*num_queries_vis_abstractors_slow))
+        image_sizes = list(chain(*image_sizes))
+        is_videos = list(chain(*is_videos))
+        first_last_frames_slows = list(chain(*first_last_frames_slows))
+
+        # Use slowfast mode if there's at least one visual token count greater than 0 in num_queries_vis_abstractors_slow
+        use_slowfast = any([num_query > 0 for num_query in num_queries_vis_abstractors_slow])
+        num_grids = [pixel_value.shape[0] for pixel_value in chain(*pixel_values)]
+        num_grids = [0] + num_grids
+        group_ids = []
+
+        if use_slowfast:
+            new_num_grids = [num_grids[0]]
+            new_num_queries = []
+            new_image_sizes = []
+            new_is_videos = []
+
+            # When using slowfast, split more finely
+            # 0th local grid is slow frame, remaining local grids are fast frames
+            for (
+                num_query,
+                num_query_slow,
+                num_grid,
+                image_size,
+                is_video,
+                first_last_frames_slow,
+                is_first_image,
+                is_last_image,
+            ) in zip(
+                num_queries_vis_abstractors,
+                num_queries_vis_abstractors_slow,
+                num_grids[1:],
+                image_sizes,
+                is_videos,
+                first_last_frames_slows,
+                is_first_images,
+                is_last_images,
+            ):
+
+                if not first_last_frames_slow and num_query_slow > 0:  # Process all image in slowfast mode
+                    assert is_video  # slowfast mode is only applied to videos
+
+                    this_group_ids = [group_ids[-1][-1] + 1 if group_ids else 0]
+
+                    # slow frame (first grid)
+                    new_num_grids.append(new_num_grids[-1] + 1)
+                    new_num_queries.append(num_query_slow)
+                    new_image_sizes.append(image_size)
+                    new_is_videos.append(is_video)
+
+                    if num_grid >= 2:
+                        # fast frames
+                        new_num_grids.append(new_num_grids[-1] + num_grid - 1)
+                        new_num_queries.append(num_query)
+                        new_image_sizes.append(image_size)
+                        new_is_videos.append(is_video)
+                        this_group_ids.append(this_group_ids[-1] + 1)
+
+                    group_ids.append(this_group_ids)
+                elif (
+                    first_last_frames_slow and num_query_slow > 0 and (is_first_image or is_last_image)
+                ):  # Process only first/last image in slowfast mode
+                    # Case for special treatment of first/last frames in slow mode
+                    assert is_video  # slowfast mode is only applied to videos
+
+                    this_group_ids = [group_ids[-1][-1] + 1 if group_ids else 0]
+
+                    if num_grid == 1:
+                        # Simply process with slow since there's only one grid
+                        new_num_grids.append(new_num_grids[-1] + 1)
+                        new_num_queries.append(num_query_slow)
+                        new_image_sizes.append(image_size)
+                        new_is_videos.append(is_video)
+
+                    if num_grid >= 2:
+                        # Special treatment for first or last grid depending on is_first_image or is_last_image
+
+                        if is_first_image:  # includes both first and last
+                            # slow frame (first grid)
+                            new_num_grids.append(new_num_grids[-1] + 1)
+                            new_num_queries.append(num_query_slow)
+                            new_image_sizes.append(image_size)
+                            new_is_videos.append(is_video)
+                            # fast frames
+                            new_num_grids.append(new_num_grids[-1] + num_grid - 1)
+                            new_num_queries.append(num_query)
+                            new_image_sizes.append(image_size)
+                            new_is_videos.append(is_video)
+                            this_group_ids.append(this_group_ids[-1] + 1)
+                        elif is_last_image:
+                            # fast frames
+                            new_num_grids.append(new_num_grids[-1] + num_grid - 1)
+                            new_num_queries.append(num_query)
+                            new_image_sizes.append(image_size)
+                            new_is_videos.append(is_video)
+                            # slow frame (last grid)
+                            new_num_grids.append(new_num_grids[-1] + 1)
+                            new_num_queries.append(num_query_slow)
+                            new_image_sizes.append(image_size)
+                            new_is_videos.append(is_video)
+                            this_group_ids.append(this_group_ids[-1] + 1)
+                        else:
+                            raise Exception("This case should not be reached.")
+                    group_ids.append(this_group_ids)
+                else:
+                    # Not in slowfast mode, so reduce all by num_query (fast)
+                    new_num_grids.append(new_num_grids[-1] + num_grid)
+                    new_num_queries.append(num_query)
+                    new_image_sizes.append(image_size)
+                    new_is_videos.append(is_video)
+
+                    start_group_id = group_ids[-1][-1] + 1 if group_ids else 0
+                    group_ids.append([start_group_id])
+
+            num_grids = new_num_grids
+            num_queries_vis_abstractors = new_num_queries
+            image_sizes = new_image_sizes
+            is_videos = new_is_videos
+        else:
+            num_grids = [sum(num_grids[:i]) for i in range(1, len(num_grids) + 1)]
+            group_ids = [[group_id] for group_id in range(len(is_videos))]
+
+        return num_queries_vis_abstractors, num_grids, image_sizes, is_videos, group_ids
+
+
+def load_state_dict_into_model(model_to_load, state_dict, strict=True, start_prefix=""):
+    # from https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/modeling_utils.py#L517
+    # Convert old format to new format if needed from a PyTorch state_dict
+    old_keys = []
+    new_keys = []
+    for key in state_dict.keys():
+        new_key = None
+        if "gamma" in key:
+            new_key = key.replace("gamma", "weight")
+        if "beta" in key:
+            new_key = key.replace("beta", "bias")
+        if new_key:
+            old_keys.append(key)
+            new_keys.append(new_key)
+    for old_key, new_key in zip(old_keys, new_keys):
+        state_dict[new_key] = state_dict.pop(old_key)
+
+    # copy state_dict so _load_from_state_dict can modify it
+    metadata = getattr(state_dict, "_metadata", None)
+    state_dict = state_dict.copy()
+    if metadata is not None:
+        state_dict._metadata = metadata
+
+    error_msgs = []
+
+    # PyTorch's `_load_from_state_dict` does not copy parameters in a module's descendants
+    # so we need to apply the function recursively.
+    def load(module: nn.Module, state_dict, prefix=""):
+        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+        args = (state_dict, prefix, local_metadata, strict, [], [], error_msgs)
+        # Parameters of module and children will start with prefix. We can exit early if there are none in this
+        # state_dict
+        if len([key for key in state_dict if key.startswith(prefix)]) > 0:
+            if is_deepspeed_zero3_enabled():
+                import deepspeed
+
+                # In sharded models, each shard has only part of the full state_dict, so only gather
+                # parameters that are in the current state_dict.
+                named_parameters = dict(module.named_parameters(prefix=prefix[:-1], recurse=False))
+                params_to_gather = [named_parameters[k] for k in state_dict.keys() if k in named_parameters]
+                if len(params_to_gather) > 0:
+                    # because zero3 puts placeholders in model params, this context
+                    # manager gathers (unpartitions) the params of the current layer, then loads from
+                    # the state dict and then re-partitions them again
+                    with deepspeed.zero.GatheredParameters(params_to_gather, modifier_rank=0):
+                        if torch.distributed.get_rank() == 0:
+                            module._load_from_state_dict(*args)
+            else:
+                module._load_from_state_dict(*args)
+
+        for name, child in module._modules.items():
+            if child is not None:
+                load(child, state_dict, prefix + name + ".")
+
+    load(model_to_load, state_dict, prefix=start_prefix)
+    # Delete `state_dict` so it could be collected by GC earlier. Note that `state_dict` is a copy of the argument, so
+    # it's safe to delete it.
+    del state_dict
+
+    return error_msgs
+
+
+class HCXVisionCAbstractor(nn.Module):
+    """
+    This module is based on C-Abstractor, whose license is under apache-2.0.
+    You can check the original code at https://github.com/khanrc/honeybee/blob/main/honeybee/projectors/projectors.py
+    and we made necessary modifications.
+    """
+
+    def __init__(
+        self,
+        num_queries: int,
+        num_input_tokens: int,
+        encoder_hidden_size: int,
+        hidden_size: int,
+        output_hidden_size: int,
+        pos_emb: bool = True,
+        prenorm: bool = False,
+    ):
+        super().__init__()
+        self.num_input_tokens = num_input_tokens
+        self.output_hidden_size = output_hidden_size
+
+        # Positional embedding
+        if pos_emb:
+            self.pos_emb = torch.nn.Parameter(torch.zeros(1, num_input_tokens, encoder_hidden_size))
+            self.pos_emb.data.normal_(mean=0.0, std=0.02)
+        else:
+            self.pos_emb = None
+
+        # (Optional) Pre-normalization layer
+        if prenorm:
+            self.prenorm = LayerNorm(encoder_hidden_size)
+        else:
+            self.prenorm = None
+
+        self.build_net(num_queries, encoder_hidden_size, hidden_size, output_hidden_size)
+        self.dtype = next(self.parameters()).dtype
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> torch.Tensor:
+        """
+        Args:
+            x: (B, L, encoder_hidden_size) tensor from the visual backbone (e.g. CLIP visual encoder), including cls token.
+        """
+        if self.prenorm is not None:
+            x = self.prenorm(x)
+
+        if self.pos_emb is not None:
+            x = x + self.pos_emb
+
+        x = self._forward(
+            x,
+            num_queries_vis_abstractors=num_queries_vis_abstractors,
+            num_grids=num_grids,
+        )  # (B, L, output_hidden_size)
+
+        return x
+
+    def _forward(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> torch.Tensor:
+        # x: [B, L, dim]
+        B, L, dim = x.shape
+        hw = int(L ** 0.5)
+        x = rearrange(x, "b (h w) d -> b d h w", h=hw, w=hw)
+
+        if num_queries_vis_abstractors is not None:
+            assert num_grids is not None
+            return self._forward_adaptive_num_query(x, num_queries_vis_abstractors, num_grids)
+
+        x = self.net(x)
+        x = rearrange(x, "b d h w -> b (h w) d")
+        x = self.readout(x)
+        return x
+
+    def _forward_adaptive_num_query(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> List[torch.Tensor]:
+        # self.net is consisted by 3 layers (s1, sampler, s2)
+        assert len(self.net) == 3
+
+        x = self.net[0](x)  # s1
+        new_x = []
+        for i, num_queries in enumerate(num_queries_vis_abstractors):
+            hw = int(num_queries**0.5)
+            sampler = nn.AdaptiveAvgPool2d((hw, hw))
+            out = sampler(x[num_grids[i]:num_grids[i + 1], :])
+            out = self.net[2](out)  # s2
+
+            out = rearrange(out, "b d h w -> b (h w) d")
+            out = self.readout(out)
+
+            new_x.append(out)
+        return new_x
+
+    def build_net(
+        self,
+        n_queries: int,
+        encoder_hidden_size: int,
+        hidden_size: int,
+        output_hidden_size: int,
+        depth: int = 3,
+        mlp_depth: int = 2,
+    ):
+        assert (n_queries ** 0.5).is_integer(), f"n_queries must be square number. n_queries: {n_queries}"
+        hw = int(n_queries ** 0.5)
+
+        # RegBlock = ResBlock + SE
+        RegBlock = partial(
+            RegStage,
+            stride=1,
+            dilation=1,
+            act_layer=nn.SiLU,
+            norm_layer=LayerNorm2d,
+        )
+
+        s1 = RegBlock(
+            depth,
+            encoder_hidden_size,
+            hidden_size,
+        )
+        sampler = nn.AdaptiveAvgPool2d((hw, hw))
+        s2 = RegBlock(
+            depth,
+            hidden_size,
+            hidden_size,
+        )
+
+        self.net = nn.Sequential(s1, sampler, s2)
+        self.readout = self.build_mlp(mlp_depth, hidden_size, output_hidden_size)
+
+    def build_mlp(
+        self,
+        depth: int,
+        hidden_size: int,
+        output_hidden_size: int,
+    ):
+        layers = [nn.Linear(hidden_size, output_hidden_size)]
+        for _ in range(1, depth):
+            layers.append(nn.SiLU())
+            layers.append(nn.Linear(output_hidden_size, output_hidden_size))
+        return nn.Sequential(*layers)
+
+def load_sharded_checkpoint(
+    model, folder, pick_prefix="", replace_prefix_list=[], replace_prefix_dict={}, print_info=True
+):
+    if folder is None:
+        return {}
+
+    files = os.listdir(folder)
+
+    # find relevant files
+    pytorch_bin_files = [file for file in files if file.startswith("pytorch_model") and file.endswith(".bin")]
+    safetensor_files = [file for file in files if file.endswith(".safetensors")]
+    shard_index_file = [file for file in files if file.endswith(".index.json")]
+
+    # check if sharded
+    index_present = len(shard_index_file) > 0
+    index_file = os.path.join(folder, shard_index_file[0]) if index_present else []
+
+    # check if safetensor
+    is_safetensor = len(safetensor_files) > 0
+
+    model_keys = model.state_dict().keys()
+
+    if is_safetensor:
+        from safetensors.torch import load_file
+
+        load_function = load_file
+        shard_files = safetensor_files
+    else:
+        load_function = partial(torch.load, map_location="cpu")
+        shard_files = pytorch_bin_files
+
+    # sharded case
+    if index_present:
+        with open(index_file, "r", encoding="utf-8") as f:
+            index = json.load(f)
+        loaded_keys = index["weight_map"].keys()
+        if pick_prefix:
+            loaded_keys = [k[len(pick_prefix) :] for k in loaded_keys if k.startswith(pick_prefix)]
+        if replace_prefix_list:
+            for rep_prefix in replace_prefix_list:
+                loaded_keys = [k[len(rep_prefix) :] if k.startswith(rep_prefix) else k for k in loaded_keys]
+        if replace_prefix_dict:
+            for rep_prefix in replace_prefix_dict:
+                loaded_keys = [
+                    k.replace(rep_prefix, replace_prefix_dict[rep_prefix]) if k.startswith(rep_prefix) else k
+                    for k in loaded_keys
+                ]
+
+    for i, shard_file in enumerate(shard_files):
+        state_dict = load_function(os.path.join(folder, shard_file))
+
+        # if pick_prefix, use only pick
+        if pick_prefix:
+            state_dict = {k[len(pick_prefix) :]: v for k, v in state_dict.items() if k.startswith(pick_prefix)}
+
+        for rep_prefix in replace_prefix_list:
+            state_dict = {k[len(rep_prefix) :] if k.startswith(rep_prefix) else k: v for k, v in state_dict.items()}
+
+        for rep_prefix in replace_prefix_dict:
+            state_dict = {
+                k.replace(rep_prefix, replace_prefix_dict[rep_prefix]) if k.startswith(rep_prefix) else k: v
+                for k, v in state_dict.items()
+            }
+
+        if is_deepspeed_zero3_enabled():
+            # torch.distributed.barrier()
+            rank = torch.distributed.get_rank()
+            print(f"# [info] ZeRo3 - load sharded no {i}, rank {rank}")
+            load_state_dict_into_model(model, state_dict, strict=False)
+        elif is_fsdp_enabled():
+            if is_local_dist_rank_0():
+                model.load_state_dict(state_dict, strict=False)
+        else:
+            model.load_state_dict(state_dict, strict=False)
+        # Make sure memory is freed before we load the next state dict.
+
+        if not index_present:
+            loaded_keys = state_dict.keys()
+
+        del state_dict
+        gc.collect()
+
+    # missing keys
+    missing_keys = [key for key in model_keys if key not in loaded_keys]
+    unexpected_keys = [key for key in loaded_keys if key not in model_keys]
+
+    if get_rank() == 0 and print_info:
+        print(f"[info] missing_keys: {missing_keys}")
+        print(f"[info] unexpected_keys: {unexpected_keys}")
+
+    return {"missing_keys": missing_keys, "unexpected_keys": unexpected_keys}

preprocessor.py

@@ -0,0 +1,1583 @@
+import base64
+import copy
+import io
+import math
+import os
+import uuid
+from typing import Dict, List, Optional, Union
+from urllib.parse import urlparse
+
+import av
+import cv2
+import numpy as np
+import requests
+import torch
+from decord import VideoReader, cpu
+from PIL import Image, UnidentifiedImageError
+from transformers.image_processing_utils import (
+    BaseImageProcessor,
+    BatchFeature,
+    get_size_dict,
+)
+from transformers.image_transforms import (
+    convert_to_rgb,
+    get_resize_output_image_size,
+    resize,
+    to_channel_dimension_format,
+)
+from transformers.image_utils import (
+    OPENAI_CLIP_MEAN,
+    OPENAI_CLIP_STD,
+    ChannelDimension,
+    ImageInput,
+    PILImageResampling,
+    get_image_size,
+    infer_channel_dimension_format,
+    is_scaled_image,
+    make_list_of_images,
+    to_numpy_array,
+    valid_images,
+)
+from transformers.utils import TensorType, logging
+
+logger = logging.get_logger(__name__)
+
+
+def determine_possible_resolutions(anyres: bool, max_num_grids: int, grid_size: int, use_1x1_grid: bool = False):
+    """
+    Finds and returns possible resolution combinations with a total number of grids less than or equal to max_num_grids.
+
+    For example, if max_num_grids is 4, the possible grid combinations are:
+    [1x1, 1x2, 1x3, 1x4, 2x1, 2x2, 3x1, 4x1], and the resolutions are calculated accordingly.
+
+    Example:
+        >>> possible_resolutions = determine_possible_resolutions(anyres=True, max_num_grids=4, grid_size=336)
+        >>> print(possible_resolutions)
+        [[336, 336], [336, 672], [336, 1008], [336, 1344], [672, 336], [672, 672], [1008, 336], [1344, 336]]
+
+    Args:
+        anyres (bool): Whether to allow any resolution combinations up to the maximum grid count.
+        max_num_grids (int): The maximum number of grids allowed (height x width must be ≤ this value).
+        grid_size (int): The size of each grid in pixels (e.g., 336).
+        use_1x1_grid (bool, optional): Whether to include the 1x1 grid as a valid resolution. Defaults to False.
+
+    Returns:
+        List[List[int]]: A list of possible [height, width] resolution pairs.
+    """
+    possible_resolutions = []
+    if anyres:
+        assert max_num_grids > 0
+        for i in range(1, max_num_grids + 1):
+            for j in range(1, max_num_grids + 1):
+                if i == 1 and j == 1 and not use_1x1_grid:
+                    continue
+                if i * j <= max_num_grids:
+                    possible_resolutions.append([i, j])
+
+        possible_resolutions = [[ys * grid_size, xs * grid_size] for ys, xs in possible_resolutions]
+
+    return possible_resolutions
+
+
+def divide_to_grids(image: np.array, grid_size: int, input_data_format=None) -> List[np.array]:
+    """
+    Divides a local image into grids of size (grid_size x grid_size).
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        grid_size (int): The size (in pixels) of each square grid.
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        List[np.array]: A list of image patches, each of size (grid_size x grid_size).
+    """
+    grids = []
+    height, width = get_image_size(image, channel_dim=input_data_format)
+    for i in range(0, height, grid_size):
+        for j in range(0, width, grid_size):
+            if input_data_format == ChannelDimension.LAST:
+                grid = image[i : i + grid_size, j : j + grid_size]
+            else:
+                grid = image[:, i : i + grid_size, j : j + grid_size]
+            grids.append(grid)
+
+    return grids
+
+
+def pad(
+    image: np.array,
+    target_size: tuple,
+    background_color=(127, 127, 127),
+    input_data_format=None,
+) -> np.array:
+    """
+    Pads the input image on the sides (top/bottom and left/right) to match the target height and width.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        target_size (tuple): Target size as (target_height, target_width).
+        background_color (tuple, optional): RGB color value used for padding. Defaults to (127, 127, 127).
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        np.array: The padded image with the specified target size.
+    """
+    target_height, target_width = target_size
+    height, width = get_image_size(image, channel_dim=input_data_format)
+
+    # result = np.ones((target_height, target_width, image.shape[2]), dtype=image.dtype) * background_color
+    result = np.empty((target_height, target_width, image.shape[2]), dtype=image.dtype)
+    for i in range(image.shape[2]):
+        result[..., i].fill(background_color[i])
+
+    paste_x = (target_width - width) // 2
+    paste_y = (target_height - height) // 2
+
+    result[paste_y : paste_y + height, paste_x : paste_x + width, :] = image
+
+    return result
+
+
+def expand2square(
+    image: np.array,
+    bboxes_dict=None,
+    background_color=(127, 127, 127),
+    input_data_format=None,
+) -> np.array:
+    """
+    Expands the input image to a square shape by placing it at the center of a new square canvas,
+    with padding added to the shorter side (either top/bottom or left/right).
+
+    The image is always centered on the new canvas, and padding is applied symmetrically.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        bboxes_dict (dict, optional): A dictionary of bounding boxes, where each value is an NDArray of shape (N, 4, 2)
+            with box coordinates in the format [[xtl, ytl], [xtr, ytr], [xbr, ybr], [xbl, ybl]].
+            Supports multiple categories (e.g., "ocr", "html") simultaneously.
+        background_color (tuple, optional): RGB color to fill the padding area. Defaults to (127, 127, 127).
+        input_data_format (optional): Optional format specifier for image data (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        np.array: A square-shaped image with the original image centered and padded as needed.
+
+    Example:
+        >>> _img = np.ones((80, 100), dtype=np.uint8) * 100
+        >>> _bboxes_dict = {"words": np.array([[[10, 10], [20, 10], [20, 20], [10, 20]],
+        ...                                    [[30, 30], [40, 30], [40, 40], [30, 40]]])}
+        >>> _img, _bboxes_dict = expand2square(_img, _bboxes_dict, (255, 255, 255))
+        >>> _img.shape
+        (100, 100)
+        >>> guessed_ocr_bboxes = np.array([[[20, 10], [30, 10], [30, 20], [20, 20]],
+        ...                                [[40, 30], [50, 30], [50, 40], [40, 40]]])
+        >>> np.testing.assert_array_almost_equal(_bboxes_dict["words"], guessed_ocr_bboxes) is None
+        True
+    """
+    height, width = get_image_size(image, channel_dim=input_data_format)
+    if width == height:
+        return image, bboxes_dict
+    elif width > height:
+        # result = np.ones((width, width, image.shape[2]), dtype=image.dtype) * background_color
+        result = np.empty((width, width, image.shape[2]), dtype=image.dtype)
+        for i in range(image.shape[2]):
+            result[..., i].fill(background_color[i])
+
+        result[(width - height) // 2 : (width - height) // 2 + height, :] = image
+        if bboxes_dict is not None:
+            for key in bboxes_dict:
+                bboxes_dict[key][:, :, 1] += (width - height) // 2
+        return result, bboxes_dict
+    else:
+        # result = np.ones((height, height, image.shape[2]), dtype=image.dtype) * background_color
+        result = np.empty((height, height, image.shape[2]), dtype=image.dtype)
+        for i in range(image.shape[2]):
+            result[..., i].fill(background_color[i])
+
+        result[:, (height - width) // 2 : (height - width) // 2 + width] = image
+        if bboxes_dict is not None:
+            for key in bboxes_dict:
+                bboxes_dict[key][:, :, 0] += (height - width) // 2
+        return result, bboxes_dict
+
+
+def resize_longside(
+    image: np.array,
+    size: int,
+    resample: PILImageResampling = PILImageResampling.BICUBIC,
+    data_format: Optional[Union[str, ChannelDimension]] = None,
+    input_data_format: Optional[Union[str, ChannelDimension]] = None,
+):
+    """
+    Resizes the image so that its longer side matches the specified size, maintaining the original aspect ratio.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        size (int): Target size for the longer side of the image.
+        resample (PILImageResampling, optional): Resampling method to use during resizing. Defaults to BICUBIC.
+        data_format (str or ChannelDimension, optional): Output data format (e.g., "channels_first" or "channels_last").
+        input_data_format (str or ChannelDimension, optional): Input data format of the image.
+
+    Returns:
+        np.array: The resized image with its aspect ratio preserved.
+    """
+    height, width = get_image_size(image, channel_dim=input_data_format)
+
+    if width == height:
+        target_height, target_width = size, size
+    elif width > height:
+        target_width = size
+        target_height = math.ceil(height / width * size)
+    else:
+        target_width = math.ceil(width / height * size)
+        target_height = size
+
+    return resize(
+        image,
+        size=(target_height, target_width),
+        resample=resample,
+        data_format=data_format,
+        input_data_format=input_data_format,
+    )
+
+
+def select_best_resolution(original_size: tuple, possible_resolutions: list) -> tuple:
+    """
+    Selects the best-fit resolution from a list of possible resolutions based on the original image size.
+
+    This function, adapted from LLaVA-Next
+    (https://github.com/huggingface/transformers/blob/v4.40.2/src/transformers/models/llava_next/image_processing_llava_next.py),
+    evaluates each resolution by computing its effective and wasted area compared to the original size.
+    The optimal resolution is the one that maximizes the effective area while minimizing unused (wasted) space.
+
+    Args:
+        original_size (tuple): The original image size in the format (height, width).
+        possible_resolutions (list): A list of candidate resolutions in the format [(height1, width1), (height2, width2), ...].
+
+    Returns:
+        tuple: The best-fit resolution in the format (height, width).
+    """
+    original_height, original_width = original_size
+    best_fit = None
+    max_effective_resolution = 0
+    min_wasted_resolution = float("inf")
+
+    for height, width in possible_resolutions:
+        scale = min(width / original_width, height / original_height)
+        downscaled_width, downscaled_height = int(original_width * scale), int(original_height * scale)
+        effective_resolution = min(downscaled_width * downscaled_height, original_width * original_height)
+        wasted_resolution = (width * height) - effective_resolution
+
+        if effective_resolution > max_effective_resolution or (
+            effective_resolution == max_effective_resolution and wasted_resolution < min_wasted_resolution
+        ):
+            max_effective_resolution = effective_resolution
+            min_wasted_resolution = wasted_resolution
+            best_fit = (height, width)
+
+    return best_fit
+
+
+def _get_local_grids_output_size(image: np.array, target_resolution: tuple, input_data_format=None):
+    """
+    Computes the number of local grids (patches) along the height and width when resizing an image
+    to the target resolution.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        target_resolution (tuple): Target resolution in the format (target_height, target_width).
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        tuple: A tuple (grid_h, grid_w) representing the number of grids along the height and width.
+    """
+    original_height, original_width = get_image_size(image, channel_dim=input_data_format)
+    target_height, target_width = target_resolution
+
+    scale_w = target_width / original_width
+    scale_h = target_height / original_height
+
+    if scale_w < scale_h:
+        new_width = target_width
+        new_height = min(math.ceil(original_height * scale_w), target_height)
+    else:
+        new_height = target_height
+        new_width = min(math.ceil(original_width * scale_h), target_width)
+
+    return new_height, new_width
+
+
+def determine_anyres_num_vision_patches(
+    num_grids,
+    image_size,
+    grid_size,
+    patch_size,
+    possible_resolutions,
+    anyres=False,
+    unpad=True,
+    num_queries_vis_abstractor=0,
+    num_queries_vis_abstractor_slow=0,
+    is_video=False,
+    first_last_frames_slow=False,  # sample-wise option
+    is_first_or_last_frames=False,  # grid-wise option
+):
+    """
+    Computes the number of visual tokens (patches) based on image resolution, grid configuration, and patch size.
+
+    This function supports both fixed-size and any-resolution settings, as well as video-specific configurations
+    such as handling slow frames and frame position flags.
+
+    Args:
+        num_grids (int): Number of grids per image (e.g., 1 for 1x1, 4 for 2x2, etc.).
+        image_size (tuple): The original image size as (height, width).
+        grid_size (int): Size of each grid in pixels (e.g., 336).
+        patch_size (int): Size of each vision patch (e.g., 14 for ViT models).
+        possible_resolutions (list): List of possible resolution tuples [(h1, w1), (h2, w2), ...].
+        anyres (bool, optional): Whether to use any-resolution mode. Defaults to False.
+        unpad (bool, optional): Whether to unpad the image before computing patches. Defaults to True.
+        num_queries_vis_abstractor (int, optional): Number of query tokens for vision abstractor (fast path).
+        num_queries_vis_abstractor_slow (int, optional): Number of query tokens for vision abstractor (slow path).
+        is_video (bool, optional): Whether the input is a video. Defaults to False.
+        first_last_frames_slow (bool, optional): Whether to treat first/last video frames as "slow". Defaults to False.
+        is_first_or_last_frames (bool, optional): Whether current grid corresponds to first/last frame. Defaults to False.
+
+    Returns:
+        int: Total number of visual tokens (patches) after processing.
+    """
+    if not anyres:
+        return num_queries_vis_abstractor if num_queries_vis_abstractor > 0 else (grid_size // patch_size) ** 2
+
+    if num_queries_vis_abstractor > 0:
+        num_patch_per_grid = int(num_queries_vis_abstractor**0.5)
+    else:
+        num_patch_per_grid = grid_size // patch_size
+
+    num_global_per_grid = num_patch_per_grid
+
+    # In anyres mode, a global image is included, so there are always at least 2 grids.
+    # However, for video inputs, there is no global image, so it's possible to have only 1 grid.
+    # Therefore, the assertion below is commented out:
+    # assert num_grids > 1
+
+    # Compute the number of vision patches.
+    height, width = select_best_resolution(image_size, possible_resolutions)
+
+    num_patch_height = (height // grid_size) * num_patch_per_grid
+    num_patch_width = (width // grid_size) * num_patch_per_grid
+
+    # local images
+    if unpad:
+        original_height, original_width = image_size
+
+        original_aspect_ratio = original_width / original_height
+        current_aspect_ratio = num_patch_width / num_patch_height
+
+        if original_aspect_ratio > current_aspect_ratio:
+            scale_factor = num_patch_width / original_width
+            new_height = int(original_height * scale_factor)
+            padding = (num_patch_height - new_height) // 2
+            num_patch_height = num_patch_height - padding * 2
+        else:
+            scale_factor = num_patch_height / original_height
+            new_width = int(original_width * scale_factor)
+            padding = (num_patch_width - new_width) // 2
+            num_patch_width = num_patch_width - padding * 2
+
+        num_patches = num_patch_width * num_patch_height + num_patch_height
+    else:
+        num_patches = num_patch_width * num_patch_height
+
+    # In the "slow" strategy, when applying to first and last frames only, it is applied exclusively to those two frames.
+    if num_queries_vis_abstractor_slow > 0:
+        if first_last_frames_slow:
+            if is_first_or_last_frames:
+                num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
+        else:
+            num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
+        # The slowfast feature is only applicable when unpad is set to False.
+        assert unpad is False
+
+    # Global image is not included for video inputs.
+    if not is_video:
+        num_patches += num_global_per_grid**2
+
+    return num_patches
+
+
+class HCXVisionProcessor(BaseImageProcessor):
+    r"""
+    Constructs a VLM image processor.
+
+    This processor is based on [`CLIPImageProcessor`] and incorporates additional techniques
+    for handling high-resolution images, such as flexible resolution support (`anyres`), unpadding,
+    square padding, and multi-grid patching strategies.
+
+    Args:
+        do_resize (bool): Whether to resize the image.
+        size (Dict[str, int], optional): Target size for resizing, typically with keys `"height"` and `"width"`.
+        anyres (bool): Whether to enable the any-resolution (`anyres`) feature, which allows flexible resolution handling via grid division.
+        unpad (bool): When `anyres` is enabled, whether to remove visual tokens corresponding to pure padding regions.
+        max_num_grids (int): Maximum number of grids allowed per image.
+        max_image_cnt (int): Maximum number of images that can be processed at once (used for batching).
+        num_queries_vis_abstractor (int): Number of visual query tokens per grid when using a visual resampler (e.g., Perceiver).
+        num_queries_vis_abstractor_video_fast (int): Number of visual queries for fast-path video frames.
+        num_queries_vis_abstractor_video_slow (int): Number of visual queries for slow-path video frames (e.g., first/last).
+        possible_resolutions (List): List of allowed resolution pairs when `anyres` is enabled. Example: [[336, 336], [336, 672], [672, 336]].
+        patch_size (int): Patch size for the Vision Transformer (ViT).
+        pad_to_square (bool): Whether to pad images to a square shape. If `False`, a center crop is applied to fit ViT input.
+        resample (PILImageResampling): Resampling method to use for resizing. Default is `BICUBIC`.
+        do_center_crop (bool): Whether to apply center cropping.
+        crop_size (Dict[str, int], optional): Size for center cropping.
+        do_rescale (bool): Whether to rescale pixel values.
+        rescale_factor (float or int): Factor to use for rescaling pixel values (typically `1/255`).
+        do_normalize (bool): Whether to normalize pixel values using `image_mean` and `image_std`.
+        image_mean (float or List[float], optional): Mean values for normalization. Can be a single float or list of floats per channel.
+        image_std (float or List[float], optional): Standard deviation values for normalization. Can be a single float or list of floats per channel.
+        do_convert_rgb (bool): Whether to convert the input image to RGB.
+        first_last_frames_slow (bool): Whether to treat the first and last frames of a video as “slow path” (processed differently).
+
+    Attributes:
+        model_input_names (List[str]): Names of the expected model inputs. Defaults to `["pixel_values"]`.
+    """
+
+    model_input_names = ["pixel_values"]
+
+    def __init__(
+        self,
+        do_resize: bool = True,
+        size: Dict[str, int] = None,
+        anyres: bool = False,
+        unpad: bool = False,
+        max_num_grids: int = 9,
+        max_image_cnt: int = 12,
+        num_queries_vis_abstractor: int = 0,
+        num_queries_vis_abstractor_video_fast: int = 0,
+        num_queries_vis_abstractor_video_slow: int = 0,
+        possible_resolutions: List = [],
+        patch_size: int = 14,
+        pad_to_square: bool = True,
+        resample: PILImageResampling = PILImageResampling.BICUBIC,
+        do_center_crop: bool = True,
+        crop_size: Dict[str, int] = None,
+        do_rescale: bool = True,
+        rescale_factor: Union[int, float] = 1 / 255,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = True,
+        first_last_frames_slow: bool = False,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        size = size if size is not None else {"shortest_edge": 512}
+        size = get_size_dict(size, default_to_square=False)
+        crop_size = crop_size if crop_size is not None else {"height": 512, "width": 512}
+        crop_size = get_size_dict(crop_size, default_to_square=True, param_name="crop_size")
+
+        self.do_resize = do_resize
+        self.size = size
+        self.anyres = anyres
+        self.unpad = unpad
+        self.max_num_grids = max_num_grids
+        self.max_image_cnt = max_image_cnt
+        self.num_queries_vis_abstractor = num_queries_vis_abstractor
+        self.num_queries_vis_abstractor_video_fast = num_queries_vis_abstractor_video_fast
+        self.num_queries_vis_abstractor_video_slow = num_queries_vis_abstractor_video_slow
+        self.possible_resolutions = [_resolution for _resolution in possible_resolutions]
+        self.patch_size = patch_size
+        self.pad_to_square = pad_to_square
+        self.resample = resample
+        self.do_center_crop = do_center_crop
+        self.crop_size = crop_size
+        self.do_rescale = do_rescale
+        self.rescale_factor = rescale_factor
+        self.do_normalize = do_normalize
+        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
+        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
+        self.do_convert_rgb = do_convert_rgb
+        self.first_last_frames_slow = first_last_frames_slow
+
+        assert self.crop_size["height"] == self.crop_size["width"]
+
+    def resize(
+        self,
+        image: np.ndarray,
+        size: Dict[str, int],
+        resample: PILImageResampling = PILImageResampling.BICUBIC,
+        data_format: Optional[Union[str, ChannelDimension]] = None,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+        **kwargs,
+    ) -> np.ndarray:
+        """
+        Resizes the input image to the specified target size.
+
+        Args:
+            image (np.ndarray): The input image to resize.
+            size (Dict[str, int]): A dictionary specifying the target size with keys `"height"` and `"width"`.
+            resample (PILImageResampling, optional): The resampling filter to use. Defaults to `BICUBIC`.
+            data_format (str or ChannelDimension, optional): The desired output data format (e.g., "channels_last").
+            input_data_format (str or ChannelDimension, optional): The input data format of the image.
+            **kwargs: Additional keyword arguments, if any.
+
+        Returns:
+            np.ndarray: The resized image as a NumPy array.
+        """
+        default_to_square = True
+        if "shortest_edge" in size:
+            size = size["shortest_edge"]
+            default_to_square = False
+        elif "height" in size and "width" in size:
+            size = (size["height"], size["width"])
+        else:
+            raise ValueError("Size must contain either 'shortest_edge' or 'height' and 'width'.")
+
+        output_size = get_resize_output_image_size(
+            image,
+            size=size,
+            default_to_square=default_to_square,
+            input_data_format=input_data_format,
+        )
+
+        return resize(
+            image,
+            size=output_size,
+            resample=resample,
+            data_format=data_format,
+            input_data_format=input_data_format,
+            **kwargs,
+        )
+
+    def _preprocess(
+        self,
+        images: ImageInput,
+        do_resize: bool = None,
+        size: Dict[str, int] = None,
+        resample: PILImageResampling = None,
+        do_center_crop: bool = None,
+        crop_size: int = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+    ) -> Image.Image:
+        """
+        Applies a sequence of preprocessing operations to the input image(s), including resizing, cropping, rescaling,
+        normalization, and format conversion.
+
+        This method is typically used internally to prepare images for model input.
+
+        Args:
+            images (ImageInput): A single image or a batch of images to preprocess.
+            do_resize (bool, optional): Whether to resize the image(s).
+            size (Dict[str, int], optional): Target size for resizing, with keys `"height"` and `"width"`.
+            resample (PILImageResampling, optional): Resampling method to use for resizing.
+            do_center_crop (bool, optional): Whether to apply center cropping.
+            crop_size (int, optional): Size of the center crop (applied to both height and width).
+            do_rescale (bool, optional): Whether to rescale the image pixel values.
+            rescale_factor (float, optional): Factor to use when rescaling pixel values (e.g., 1/255).
+            do_normalize (bool, optional): Whether to normalize the image using `image_mean` and `image_std`.
+            image_mean (float or List[float], optional): Mean value(s) used for normalization.
+            image_std (float or List[float], optional): Standard deviation value(s) used for normalization.
+            data_format (ChannelDimension, optional): The desired output data format (e.g., `ChannelDimension.FIRST`).
+            input_data_format (str or ChannelDimension, optional): The format of the input image(s).
+
+        Returns:
+            Image.Image: The preprocessed image or batch of images, ready for model input.
+        """
+        images = make_list_of_images(images)
+
+        if do_resize:
+            images = [
+                self.resize(
+                    image=image,
+                    size=size,
+                    resample=resample,
+                    input_data_format=input_data_format,
+                )
+                for image in images
+            ]
+
+        if do_center_crop:
+            images = [
+                self.center_crop(image=image, size=crop_size, input_data_format=input_data_format) for image in images
+            ]
+
+        if do_rescale:
+            images = [
+                self.rescale(
+                    image=image,
+                    scale=rescale_factor,
+                    input_data_format=input_data_format,
+                )
+                for image in images
+            ]
+
+        if do_normalize:
+            images = [
+                self.normalize(
+                    image=image,
+                    mean=image_mean,
+                    std=image_std,
+                    input_data_format=input_data_format,
+                )
+                for image in images
+            ]
+
+        images = [
+            to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
+        ]
+
+        return images
+
+    def _resize_for_local_grids(
+        self,
+        image: np.array,
+        target_resolution: tuple,
+        resample,
+        input_data_format: ChannelDimension,
+    ) -> np.array:
+        """
+        Resizes the image to the given target resolution for use in local grid processing.
+
+        This function ensures that the image is properly resized to match the (height, width) specified
+        in `target_resolution`, using the provided resampling method. It supports channel-first and
+        channel-last formats based on `input_data_format`.
+
+        Args:
+            image (np.array): Input image as a NumPy array.
+            target_resolution (tuple): Target resolution as (height, width) for resizing.
+            resample: Resampling method to use (e.g., `PILImageResampling.BICUBIC`).
+            input_data_format (ChannelDimension): Format of the input image (e.g., `ChannelDimension.FIRST` or `LAST`).
+
+        Returns:
+            np.array: The resized image in NumPy array format.
+        """
+        new_height, new_width = _get_local_grids_output_size(image, target_resolution, input_data_format)
+
+        # Resize the image
+        resized_image = resize(
+            image,
+            (new_height, new_width),
+            resample=resample,
+            input_data_format=input_data_format,
+        )
+
+        return resized_image
+
+    def _pad_for_patching(
+        self,
+        image: np.array,
+        target_resolution: tuple,
+        input_data_format: ChannelDimension,
+    ) -> np.array:
+        """
+        Pads the image to match the target resolution, ensuring compatibility with patch-based models.
+
+        This is typically used to make sure the image dimensions are divisible by the patch size or to
+        meet specific model input requirements. Padding is applied symmetrically where needed.
+
+        Args:
+            image (np.array): Input image as a NumPy array.
+            target_resolution (tuple): The desired resolution after padding, in the format (height, width).
+            input_data_format (ChannelDimension): Format of the input image (e.g., `ChannelDimension.FIRST` or `LAST`).
+
+        Returns:
+            np.array: The padded image as a NumPy array.
+        """
+        target_height, target_width = target_resolution
+
+        background_color = tuple(int(x * 255) for x in self.image_mean)
+        padded_image = pad(
+            image,
+            target_size=(target_height, target_width),
+            background_color=background_color,
+            input_data_format=input_data_format,
+        )
+
+        return padded_image
+
+    def get_image_grids(
+        self,
+        image: np.array,
+        possible_resolutions,
+        grid_size: int,
+        resample: PILImageResampling,
+        data_format: ChannelDimension,
+        input_data_format: ChannelDimension,
+    ) -> List[np.array]:
+        """
+        Splits the input image into multiple local grids based on possible resolutions and grid size.
+
+        The function selects the best resolution from the provided list, resizes the image accordingly,
+        and divides it into non-overlapping grid patches of size (grid_size x grid_size). It is commonly
+        used for any-resolution (anyres) visual processing.
+
+        Args:
+            image (np.array): Input image as a NumPy array.
+            possible_resolutions (List[Tuple[int, int]]): List of allowed resolutions to choose from.
+            grid_size (int): The size of each grid patch (e.g., 336 pixels).
+            resample (PILImageResampling): Resampling method used during resizing.
+            data_format (ChannelDimension): Output data format (e.g., `ChannelDimension.FIRST`).
+            input_data_format (ChannelDimension): Input data format of the image.
+
+        Returns:
+            List[np.array]: A list of grid image patches as NumPy arrays.
+        """
+        if not isinstance(possible_resolutions, list):
+            raise ValueError("possible_resolutions must be a list of possible resolutions.")
+
+        image_size = get_image_size(image, channel_dim=input_data_format)
+        best_resolution = select_best_resolution(image_size, possible_resolutions)
+        resized_image = self._resize_for_local_grids(
+            image,
+            best_resolution,
+            resample=resample,
+            input_data_format=input_data_format,
+        )
+        padded_image = self._pad_for_patching(resized_image, best_resolution, input_data_format=input_data_format)
+        local_grids = divide_to_grids(padded_image, grid_size=grid_size, input_data_format=input_data_format)
+
+        # make sure that all patches are in the input data format
+        local_grids = [
+            to_channel_dimension_format(grid, channel_dim=data_format, input_channel_dim=input_data_format)
+            for grid in local_grids
+        ]
+
+        return local_grids
+
+    def preprocess(
+        self,
+        images: ImageInput,
+        do_resize: bool = None,
+        size: Dict[str, int] = None,
+        anyres: bool = None,
+        unpad: bool = None,
+        is_video_list: List[bool] = None,
+        possible_resolutions: List = None,
+        patch_size: int = None,
+        pad_to_square: bool = None,
+        resample: PILImageResampling = None,
+        do_center_crop: bool = None,
+        crop_size: int = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+        is_first_or_last_frames: List[bool] = False,
+    ):
+        """
+        Preprocesses images using HCXVisionProcessor.
+
+        This method prepares images for visual language models by applying resizing, padding, cropping,
+        normalization, and tokenization into visual patches. In video mode, each frame is converted to
+        a 1D sequence of patches. The `unpad` option is disabled when processing videos.
+
+        Args:
+            images (ImageInput): A single image or a batch of images (PIL, NumPy, or tensor format).
+            do_resize (bool, optional): Whether to resize the image(s).
+            size (Dict[str, int], optional): Resize target with keys `"height"` and `"width"`.
+            anyres (bool, optional): Whether to use any-resolution processing with grid splitting.
+            unpad (bool, optional): Whether to remove visual tokens that belong to padding areas (only in non-video mode).
+            is_video_list (List[bool], optional): A list indicating which inputs are video frames.
+            possible_resolutions (List, optional): List of resolution pairs allowed in `anyres` mode.
+            patch_size (int, optional): Patch size for the Vision Transformer (ViT).
+            pad_to_square (bool, optional): Whether to pad the image to a square.
+            resample (PILImageResampling, optional): Resampling method to use for resizing.
+            do_center_crop (bool, optional): Whether to apply center cropping.
+            crop_size (int, optional): Target crop size for center cropping.
+            do_rescale (bool, optional): Whether to rescale image pixel values.
+            rescale_factor (float, optional): Factor for pixel rescaling, e.g., `1/255`.
+            do_normalize (bool, optional): Whether to normalize using mean and std.
+            image_mean (float or List[float], optional): Mean value(s) for normalization.
+            image_std (float or List[float], optional): Standard deviation(s) for normalization.
+            do_convert_rgb (bool, optional): Whether to convert the image to RGB.
+            return_tensors (str or TensorType, optional): Desired output tensor type (e.g., "pt" for PyTorch).
+            data_format (ChannelDimension, optional): Output data format (e.g., `ChannelDimension.FIRST`).
+            input_data_format (str or ChannelDimension, optional): Format of the input image.
+            is_first_or_last_frames (List[bool], optional): Flags indicating whether each image is a first/last video frame.
+
+        Returns:
+            Tuple:
+                pixel_values (List[torch.Tensor]): A list of 4D image tensors ready for model input.
+                image_sizes (List[List[int]]): A list of list containing the original width and height [width, height]
+                    of each image, e.g., `[[width, height], ...]`.
+                vision_query_lengths (List[int]): A list of integers representing the number of visual tokens
+                    each image contributes to the LLM input.
+        """
+        do_resize = do_resize if do_resize is not None else self.do_resize
+        size = size if size is not None else self.size
+        size = get_size_dict(size, param_name="size", default_to_square=False)
+        anyres = anyres if anyres is not None else self.anyres
+        unpad = unpad if unpad is not None else self.unpad
+        possible_resolutions = possible_resolutions if possible_resolutions is not None else self.possible_resolutions
+        patch_size = patch_size if patch_size is not None else self.patch_size
+        pad_to_square = pad_to_square if pad_to_square is not None else self.pad_to_square
+        resample = resample if resample is not None else self.resample
+        do_center_crop = do_center_crop if do_center_crop is not None else self.do_center_crop
+        crop_size = crop_size if crop_size is not None else self.crop_size
+        crop_size = get_size_dict(crop_size, param_name="crop_size", default_to_square=True)
+        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
+        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
+        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
+        image_mean = image_mean if image_mean is not None else self.image_mean
+        image_std = image_std if image_std is not None else self.image_std
+        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
+
+        images = make_list_of_images(images)
+
+        if not valid_images(images):
+            raise ValueError(
+                "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
+                "torch.Tensor, tf.Tensor or jax.ndarray."
+            )
+
+        if do_convert_rgb:
+            images = [convert_to_rgb(image) for image in images]
+
+        # All transformations expect numpy arrays.
+        images = [to_numpy_array(image) for image in images]
+
+        if is_scaled_image(images[0]) and do_rescale:
+            logger.warning_once(
+                "It looks like you are trying to rescale already rescaled images. If the input"
+                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
+            )
+
+        if input_data_format is None:
+            # We assume that all images have the same channel dimension format.
+            input_data_format = infer_channel_dimension_format(images[0])
+
+        new_images = []
+        image_sizes = [get_image_size(image, channel_dim=input_data_format) for image in images]
+        vision_query_lengths = []
+
+        assert crop_size["height"] == crop_size["width"]
+
+        # Padding operations for the global image can become a bottleneck when the original image width or height is large.
+        # To mitigate this, the image is first resized such that the longest side is scaled proportionally based on size["shortest_edge"],
+        # and then padding is applied to reach the target dimensions.
+        if anyres:
+            anyres_global_images = copy.deepcopy(images)
+            if pad_to_square:
+                background_color = tuple(int(x * 255) for x in self.image_mean)
+                anyres_global_images = [
+                    resize_longside(
+                        copy.deepcopy(image),
+                        size["shortest_edge"],
+                        resample,
+                        input_data_format,
+                    )
+                    for image in anyres_global_images
+                ]
+                anyres_global_images = [
+                    expand2square(
+                        image,
+                        background_color=background_color,
+                        input_data_format=input_data_format,
+                    )[0]
+                    for image in anyres_global_images
+                ]
+            else:
+                anyres_global_images = [
+                    self.resize(
+                        image=image,
+                        size={
+                            "height": size["shortest_edge"],
+                            "width": size["shortest_edge"],
+                        },
+                        resample=resample,
+                        input_data_format=input_data_format,
+                    )
+                    for image in anyres_global_images
+                ]
+        else:
+            anyres_global_images = [None for _ in range(len(images))]
+            if pad_to_square:
+                background_color = tuple(int(x * 255) for x in self.image_mean)
+                images = [
+                    resize_longside(image, size["shortest_edge"], resample, input_data_format) for image in images
+                ]
+                images = [
+                    expand2square(
+                        image,
+                        background_color=background_color,
+                        input_data_format=input_data_format,
+                    )[0]
+                    for image in images
+                ]
+
+        num_queries_vis_abstractors = []
+        num_queries_vis_abstractors_slow = []
+        first_last_frames_slows = []
+
+        for image, is_video, anyres_global_image, image_size in zip(
+            images, is_video_list, anyres_global_images, image_sizes
+        ):
+            if is_video:
+                num_queries_vis_abstractor = self.num_queries_vis_abstractor_video_fast
+                num_queries_vis_abstractor_slow = self.num_queries_vis_abstractor_video_slow
+            else:
+                num_queries_vis_abstractor = self.num_queries_vis_abstractor
+                num_queries_vis_abstractor_slow = 0
+
+            num_queries_vis_abstractors.append(num_queries_vis_abstractor)
+            num_queries_vis_abstractors_slow.append(num_queries_vis_abstractor_slow)
+            first_last_frames_slows.append(self.first_last_frames_slow)
+
+            if anyres:
+                # convert image into a list of grids
+                # we intentially use the same data format as the input data format
+                image_grids = self.get_image_grids(
+                    image,
+                    possible_resolutions,
+                    grid_size=crop_size["height"],
+                    resample=resample,
+                    data_format=input_data_format,
+                    input_data_format=input_data_format,
+                )
+                # Global image (thumbnail) is not used for video inputs.
+                if not is_video:
+                    image_grids = [anyres_global_image] + image_grids
+            else:
+                image_grids = [image]
+
+            pixel_values = self._preprocess(
+                image_grids,
+                do_resize=do_resize,
+                size=size,
+                resample=resample,
+                do_center_crop=do_center_crop,
+                crop_size=crop_size,
+                do_rescale=do_rescale,
+                rescale_factor=rescale_factor,
+                do_normalize=do_normalize,
+                image_mean=image_mean,
+                image_std=image_std,
+                data_format=data_format,
+                input_data_format=input_data_format,
+            )
+
+            pixel_values = np.array(pixel_values)
+            new_images.append(pixel_values)
+
+            num_grids = pixel_values.shape[0]
+
+            vision_query_length = determine_anyres_num_vision_patches(
+                num_grids=num_grids,
+                image_size=image_size,
+                grid_size=crop_size["height"],
+                patch_size=patch_size,
+                possible_resolutions=possible_resolutions,
+                anyres=anyres,
+                unpad=False if is_video else unpad,
+                num_queries_vis_abstractor=num_queries_vis_abstractor,
+                num_queries_vis_abstractor_slow=num_queries_vis_abstractor_slow,
+                is_video=is_video,
+                first_last_frames_slow=self.first_last_frames_slow,
+                is_first_or_last_frames=self.first_last_frames_slow,
+            )
+
+            vision_query_lengths.append(vision_query_length)
+
+        data = {
+            "pixel_values": [[torch.tensor(new_image) for new_image in new_images]],
+            "image_sizes": [[[image_size[1], image_size[0]] for image_size in image_sizes]],
+            "vision_query_lengths": [vision_query_lengths],
+            "is_videos": [is_video_list],
+            "num_queries_vis_abstractors": [num_queries_vis_abstractors],
+            "num_queries_vis_abstractors_slow": [num_queries_vis_abstractors_slow],
+            "first_last_frames_slows": [first_last_frames_slows],
+        }
+
+        return BatchFeature(data=data)
+
+    def load_images_videos(self, vlm_chat):
+        """
+        Loads and prepares images or video frames from a VLM chat input.
+
+        This function parses the input `vlm_chat` object, extracts image or video sources,
+        and loads them into memory as PIL or NumPy images, ready for preprocessing.
+
+        Args:
+            vlm_chat: A VLM chat input structure containing multimodal elements
+                    (e.g., images, videos, URLs, or file paths). The format is typically a list of messages
+                    with associated media fields.
+
+        Returns:
+            List[Union[PIL.Image.Image, List[PIL.Image.Image]]]:
+                A list of loaded images. For video entries, a list of frames is returned instead of a single image.
+        """
+        vlm_chat = copy.deepcopy(vlm_chat)
+
+        new_vlm_chat = []
+        all_images = []  # images + images_from_videos
+        is_video_list = []
+
+        for line in vlm_chat:
+            if "content" in line:
+                content = line["content"]
+
+                if "image" in content:
+                    if "filename" not in content:
+                        content["filename"] = f"{uuid.uuid4().hex}.jpg"
+                    image_pil = load_image(content["image"])
+                    all_images.append(image_pil)
+                    is_video_list.append(False)
+                    new_vlm_chat.append(line)
+
+                elif "video" in content:
+                    video_bytesio = load_video_to_bytesio(content["video"])
+                    pil_img_frames, video_time_stamp = process_video(
+                        video_bytesio, self.max_num_grids, self.max_image_cnt, self.crop_size["width"]
+                    )
+                    all_images.extend(pil_img_frames)
+                    is_video_list.extend([True] * len(pil_img_frames))
+
+                    if "filename" not in content:
+                        content["filename"] = f"{uuid.uuid4().hex}.mp4"
+
+                    for i, image_time_stamp in enumerate(video_time_stamp):
+                        new_line = copy.deepcopy(line)
+                        basename, ext = os.path.splitext(content["filename"])
+                        new_line["content"]["filename"] = f"{basename}-{i}{ext}"
+                        new_line["content"]["video_time_stamp"] = image_time_stamp
+
+                        if i == len(video_time_stamp) - 1:
+                            new_line["content"]["is_final_grid"] = True
+
+                            for last_frame_target_key in ["lens_keywords", "lens_local_keywords", "speech_to_text"]:
+                                if last_frame_target_key in content:
+                                    new_line["content"][last_frame_target_key] = content[last_frame_target_key]
+
+                        new_vlm_chat.append(new_line)
+                else:
+                    new_vlm_chat.append(line)
+
+        return new_vlm_chat, all_images, is_video_list
+
+
+def process_video(video_bytesio, max_num_grids, max_image_cnt, vit_input_size):
+    """
+    Processes a video file and extracts frames suitable for vision transformer (ViT) input.
+
+    The function reads video data from a BytesIO object, extracts a limited number of frames
+    based on `max_num_grids` and `max_image_cnt`, and resizes them to the appropriate ViT input size.
+
+    Args:
+        video_bytesio (io.BytesIO): A BytesIO object containing the raw video file data.
+        max_num_grids (int): The maximum number of grids allowed (e.g., for tiling or patching).
+        max_image_cnt (int): The maximum number of frames to extract from the video.
+        vit_input_size (int): The desired input size (height and width) for the ViT model.
+
+    Returns:
+        List[np.ndarray]: A list of processed video frames as NumPy arrays, each resized to (vit_input_size, vit_input_size).
+    """
+    frames, time_interval = video_decoder(
+        video_bytesio, max_num_grids=max_num_grids, max_image_cnt=max_image_cnt, default_interval=0.4
+    )
+    pil_img_frames, video_time_stamp = combine_frames_into_images(
+        frames, time_interval, max_grid_shape=(max_num_grids, 1), vit_input_size=vit_input_size
+    )
+
+    return pil_img_frames, video_time_stamp
+
+
+def load_image(image_src):
+    """
+    Loads an image from various sources (file path, URL, base64 string, or raw bytes)
+    and returns it as a PIL Image object.
+
+    Args:
+        image_src (str or bytes): The image source. It can be:
+            - A local file path
+            - A URL
+            - A base64-encoded string
+            - Raw image bytes
+
+    Returns:
+        PIL.Image.Image: The loaded image as a PIL Image object.
+
+    Raises:
+        ValueError: If the image cannot be loaded or the format is unsupported.
+        TypeError: If the input is not of type str or bytes.
+    """
+    try:
+        # 1. If input is bytes type
+        if isinstance(image_src, bytes):
+            return Image.open(io.BytesIO(image_src))
+
+        # 2. If input is str type (path, URL, base64)
+        if isinstance(image_src, str):
+            # 2a. Check if it's a Base64 data URI format ('data:image/...')
+            if image_src.startswith("data:image"):
+                try:
+                    # Remove the 'data:image/...;base64,' part and decode
+                    header, encoded = image_src.split(",", 1)
+                    image_bytes = base64.b64decode(encoded)
+                    return Image.open(io.BytesIO(image_bytes))
+                except (ValueError, base64.binascii.Error) as e:
+                    raise ValueError(f"Invalid base64 data URI format: {e}") from e
+
+            # 2b. Check if it's a URL format ('http://' or 'https://')
+            elif image_src.startswith("http://") or image_src.startswith("https://"):
+                try:
+                    response = requests.get(image_src, stream=True, timeout=10)
+                    response.raise_for_status()  # Raise an exception for HTTP errors
+                    image_bytes = response.content
+                    return Image.open(io.BytesIO(image_bytes))
+                except requests.exceptions.RequestException as e:
+                    raise ValueError(f"Error loading image from URL '{image_src}': {e}") from e
+
+            # 2c. Assume it's a local file path
+            else:
+                return Image.open(image_src)
+
+        else:
+            raise TypeError(f"Unsupported image_src type: {type(image_src)}")
+
+    # Common exception handling
+    except FileNotFoundError:
+        raise ValueError(f"Image loading error: File not found '{image_src}'")
+    except UnidentifiedImageError:
+        raise ValueError("Image loading error: Cannot identify image file format.")
+    except IOError as e:
+        raise ValueError(f"Image loading error (I/O): {e}") from e
+    except Exception as e:
+        raise ValueError(f"Unexpected error during image loading: {e}") from e
+
+
+def load_video_to_bytesio(video_src):
+    """
+    Loads video data from various sources (file path, URL, base64 string, or raw bytes)
+    and returns an `io.BytesIO` object containing the raw video content.
+
+    Args:
+        video_src (str or bytes): The video source. Supported formats include:
+            - Local file path
+            - URL
+            - Base64-encoded data URI string
+            - Raw video bytes
+
+    Returns:
+        io.BytesIO: A `BytesIO` object containing the loaded video data.
+
+    Raises:
+        ValueError: If the video cannot be loaded due to issues such as an invalid path,
+                    URL failure, malformed base64 string, or unsupported format.
+        TypeError: If the input is not a `str` or `bytes` object.
+    """
+    video_bytes = None
+    try:
+        # 1. If input is bytes type
+        if isinstance(video_src, bytes):
+            video_bytes = video_src
+
+        # 2. If input is str type (path, URL, base64)
+        elif isinstance(video_src, str):
+            # 2a. Check if it's a Base64 data URI format ('data:video/...')
+            if video_src.startswith("data:video"):
+                try:
+                    # Remove the 'data:video/...;base64,' part and decode
+                    header, encoded = video_src.split(",", 1)
+                    video_bytes = base64.b64decode(encoded)
+                except (ValueError, base64.binascii.Error) as e:
+                    raise ValueError(f"Invalid base64 data URI format: {e}") from e
+
+            # 2b. Check if it looks like a URL
+            elif urlparse(video_src).scheme in ("http", "https"):
+                try:
+                    response = requests.get(
+                        video_src, stream=True, timeout=30
+                    )  # Increased timeout for potentially large videos
+                    response.raise_for_status()  # Raise an exception for HTTP errors (4xx or 5xx)
+                    # Read all content from the stream into bytes
+                    video_bytes = response.content
+                except requests.exceptions.MissingSchema:
+                    # If urlparse thinks it's a scheme but requests disagrees (e.g., "http:/example.com")
+                    # Treat it as a potential file path below.
+                    pass
+                except requests.exceptions.RequestException as e:
+                    raise ValueError(f"Error loading video from URL '{video_src}': {e}") from e
+
+            # 2c. Assume it's a local file path if not base64 or confirmed URL
+            if video_bytes is None:  # Only attempt file read if not already loaded as base64 or URL failed gracefully
+                # Check if it could potentially be a file path
+                # Note: This check is basic. A string like "http:/path/file" might incorrectly be treated as a path here
+                # if the requests call failed due to MissingSchema. More robust path validation could be added.
+                if (
+                    os.path.exists(video_src) or "/" in video_src or "\\" in video_src
+                ):  # Basic check if it resembles a path
+                    try:
+                        with open(video_src, "rb") as f:
+                            video_bytes = f.read()
+                    except FileNotFoundError:
+                        raise ValueError(f"Video loading error: File not found at path '{video_src}'")
+                    except IsADirectoryError:
+                        raise ValueError(f"Video loading error: Path '{video_src}' is a directory, not a file.")
+                    except IOError as e:
+                        raise ValueError(f"Video loading error (I/O) for path '{video_src}': {e}") from e
+                else:
+                    # If it's not base64, not a valid downloadable URL, and doesn't look like a path/doesn't exist
+                    raise ValueError(f"Unsupported string input format or resource not found: '{video_src}'")
+
+        # 3. If the type is unsupported
+        else:
+            raise TypeError(f"Unsupported video_src type: {type(video_src)}")
+
+        # Final check if video_bytes was successfully obtained
+        if video_bytes is None:
+            raise ValueError(f"Could not load video data from the provided source: {video_src}")
+
+        # Return the bytes wrapped in BytesIO
+        return io.BytesIO(video_bytes)
+
+    # Catch specific exceptions first for better error reporting
+    except FileNotFoundError as e:  # Should be caught above, but as a safeguard
+        raise ValueError(f"Video loading error: File not found '{video_src}'") from e
+    except requests.exceptions.RequestException as e:  # Already handled, but for clarity
+        raise ValueError(f"Video loading error (Network): {e}") from e
+    except (ValueError, TypeError) as e:  # Re-raise ValueErrors/TypeErrors raised intentionally within the try block
+        raise e
+    except Exception as e:
+        # Catch any other unexpected errors during processing
+        raise ValueError(f"Unexpected error during video loading from source '{video_src}': {e}") from e
+
+
+def video_decoder(video_bytesio, max_num_grids, max_image_cnt, default_interval=0.4):
+    """
+    Decodes video data from a BytesIO object and returns a list of extracted frames.
+
+    Args:
+        video_bytesio (io.BytesIO): A BytesIO object containing the raw video data.
+        max_num_grids (int): Maximum number of grids allowed per image. Used to determine how many frames to extract.
+        max_image_cnt (int): Maximum number of frames to extract from the video.
+        default_interval (float, optional): Default time interval (in seconds) between frames. Used when frame rate info is unavailable. TODO: make configurable.
+
+    Returns:
+        Tuple:
+            frames (List[PIL.Image.Image]): A list of extracted frames as PIL Images.
+            time_interval (float): Time interval (in seconds) between selected frames.
+    """
+    error_messages = []
+    frames = []
+
+    # 1. Try decoding the video using Decord.
+    try:
+        vr = VideoReader(video_bytesio, ctx=cpu(0), num_threads=8)
+        fps = vr.get_avg_fps()
+        play_time = len(vr) / fps
+        total_frames = len(vr)
+        frame_indices, time_interval = extract_frame_indices(
+            play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=default_interval
+        )  # Sample every 0.4 seconds; if the video is too long, apply uniform sampling instead.
+        if frame_indices is None:
+            frame_indices = range(len(vr))  # Convert all frames.
+        batch_frames = vr.get_batch(frame_indices).asnumpy()
+        frames = [Image.fromarray(frame).convert("RGB") for frame in batch_frames]
+        return frames, time_interval
+    except Exception as e:
+        print("error with decord")
+        error_messages.append(f"Decord 실패: {e}")
+
+    # 2. Fallback: Try decoding the video using PyAV.
+    try:
+        container = av.open(video_bytesio)
+        fps = container.streams.video[0].average_rate
+        play_time = len(container) / fps
+        total_frames = len(container)
+        frame_indices, time_interval = extract_frame_indices(
+            play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=default_interval
+        )  # Sample frames every 0.4 seconds. If the video is long, use uniform sampling to limit the number of frames.
+        # Even if frame_indices were assigned using Decord, reprocess them to be compatible with PyAV.
+        target_indices = None if frame_indices is None else set(frame_indices)
+        frames = []
+        for i, frame in enumerate(container.decode(video=0)):
+            if target_indices is not None and i not in target_indices:
+                continue  # Skip frames that are not in the required indices.
+            pil_frame = Image.fromarray(frame.to_ndarray(format="rgb24")).convert("RGB")
+            frames.append(pil_frame)
+        if frames:
+            return frames, time_interval
+        else:
+            raise Exception("Decoding with PyAV succeeded, but no frames were extracted.")
+    except Exception as e:
+        error_messages.append(f"PyAV failed: {e}")
+
+    # 3. Fallback: Try decoding the video using OpenCV.
+    try:
+        byte_data = np.frombuffer(video_bytesio.getvalue(), dtype=np.uint8)
+        video = cv2.imdecode(byte_data, cv2.IMREAD_UNCHANGED)
+
+        cap = cv2.VideoCapture(video)
+        total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+        fps = cap.get(cv2.CAP_PROP_FPS)
+        play_time = total_frames / fps
+        frame_indices, time_interval = extract_frame_indices(
+            play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=default_interval
+        )  # Sample frames every 0.4 seconds; if the video is too long, apply uniform sampling to limit the total number of frames.
+        if frame_indices is None:
+            frame_indices = range(total_frames)  # Convert all frames.
+
+        index_set = set(frame_indices)  # Convert to a set for faster lookup.
+        current_index = 0
+
+        while cap.isOpened():
+            ret, frame = cap.read()
+            if not ret:
+                break
+            if current_index in index_set:
+                frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)).convert("RGB"))
+            current_index += 1
+            if current_index > max(index_set):  # Stop processing once all required indices have been handled.
+                break
+
+        cap.release()
+        if frames:
+            return frames, time_interval
+    except Exception as e:
+        error_messages.append(f"OpenCV failed: {e}")
+
+    if error_messages:
+        raise Exception(f"All decoding attempts have failed.: {error_messages}")
+
+
+def convert_format_for_multi_image(img, json, convert_key_list=["words", "text", "objects", "entities"]):
+    """
+    Converts the format of image and annotation data from a single-image dataset to a multi-image dataset format.
+
+    Single-image datasets typically return a single image and its associated annotation as individual objects.
+    This function wraps them in a dictionary format used by multi-image datasets.
+
+    Args:
+        img: The input image (e.g., a PIL Image or NumPy array).
+        json: The annotation data associated with the image.
+        convert_key_list (List[str], optional): A list of keys to extract and convert from the original JSON.
+            Defaults to ["words", "text", "objects", "entities"].
+
+    Returns:
+        Tuple[Dict, Dict]:
+            - A dictionary mapping image IDs to images (e.g., {"image_0": img}).
+            - A dictionary mapping image IDs to corresponding annotation JSONs (with filtered keys).
+    """
+    is_multi_image_dataset = isinstance(img, dict)
+    if not is_multi_image_dataset:
+        img = {"00": img}
+
+        for convert_key in convert_key_list:
+            if convert_key in json:
+                json[convert_key] = {"00": json[convert_key]}
+
+        for json_key in json:
+            if "region" in json_key:
+                json[json_key] = {"00": json[json_key]}
+
+    return is_multi_image_dataset, img, json
+
+
+def convert_tags_for_video(img, json):
+    """
+    Converts <video_00> tags to <image_xx> tags based on the number of video frames.
+
+    In video datasets, annotations often use a generic <video_00> tag. This function replaces that tag
+    with frame-specific tags such as <image_00>, <image_01>, ..., <image_NN> based on the number of frames in `img`.
+
+    Args:
+        img: A list of video frames (e.g., list of PIL Images or NumPy arrays).
+        json: The annotation data containing <video_00> tags to be replaced.
+
+    Returns:
+        Dict: The updated annotation JSON with frame-specific <image_xx> tags.
+    """
+    image_tag = "".join([f"<image_{idx:02d}>" for idx in range(len(img))])
+    # image_tag = "<image_00>"  # Use this format to construct and insert image-specific tags.
+    for json_key in json:
+        if "qa_pairs" in json_key:
+            new_qa_pairs = []
+            for qa_pair in json[json_key]:
+                question = qa_pair[0]
+                # Replace <video_00> tags with corresponding <image_xx> tags.
+                question = question.replace("<video_00>", image_tag)
+                new_qa_pairs.append([question, qa_pair[1]])
+            json[json_key] = new_qa_pairs
+
+    return img, json
+
+
+def split_list(input_list, split_value):
+    """
+    Splits a list into sublists using a specified delimiter value.
+
+    Each time `split_value` is encountered in `input_list`, a new sublist is started.
+    The delimiter itself is not included in the output.
+
+    Args:
+        input_list (List[Any]): The input list to split.
+        split_value (Any): The value used as the delimiter for splitting.
+
+    Returns:
+        List[List[Any]]: A list of sublists, split by the specified delimiter.
+
+    Example:
+        >>> split_list(["a", "b", "|", "c", "d", "|", "e"], "|")
+        [['a', 'b'], ['c', 'd'], ['e']]
+    """
+    temp_list = []
+    result = []
+
+    for value in input_list:
+        if value == split_value:
+            result.append(temp_list)
+            temp_list = []
+        else:
+            temp_list.append(value)
+    result.append(temp_list)
+
+    return result
+
+
+def combine_frames_into_images(frames, time_interval, max_grid_shape=(3, 3), vit_input_size=378):
+    """
+    Combines a sequence of video frames into grid-based images and generates corresponding time range labels.
+
+    Frames are grouped and arranged into a grid (e.g., 3x3) such that each combined image contains up to
+    `max_grid_shape[0] * max_grid_shape[1]` frames. Each combined image is resized to the given ViT input size.
+
+    Args:
+        frames (List[PIL.Image.Image]): A list of frames extracted from a video.
+        time_interval (float): Time interval (in seconds) between consecutive frames.
+        max_grid_shape (Tuple[int, int], optional): The maximum grid shape as (rows, cols). Defaults to (3, 3).
+        vit_input_size (int, optional): The target size (height and width) for the Vision Transformer input. Defaults to 378.
+
+    Returns:
+        Tuple:
+            image_list (List[PIL.Image.Image]): A list of grid-combined images.
+            image_time_stamps (List[str]): A list of time span labels for each combined image,
+                e.g., ["0.00s~1.50s", "1.50s~3.00s", ...].
+    """
+    # grid_size = int(np.sqrt(max_num_grids))
+    # assert grid_size**2 == max_num_grids, "max_num_grids must be a perfect square."
+    max_num_grids = max_grid_shape[0] * max_grid_shape[1]
+    assert (
+        max_grid_shape[1] == 1
+    ), f"For video processing, decided to concatenate frames horizontally into a wide image."
+
+    # List to store the resulting combined images.
+    image_list = []
+
+    # Calculate the number of canvases needed.
+    num_frames = len(frames)
+    num_canvases = num_frames // max_num_grids
+    leftover_frames = num_frames % max_num_grids
+
+    time_stamp = 0  # second
+    image_time_stamps = []
+
+    for canvas_idx in range(num_canvases):
+        # Initialize the current canvas.
+        combined_image = Image.new(
+            "RGB", (vit_input_size * max_grid_shape[0], vit_input_size * max_grid_shape[1]), color=(0, 0, 0)
+        )
+
+        # Determine the frames to fill in the current canvas.
+        start_idx = canvas_idx * max_num_grids
+        end_idx = min(start_idx + max_num_grids, num_frames)
+
+        for idx in range(start_idx, end_idx):
+            img = frames[idx]
+
+            # Resize each frame to a square shape.
+            img_resized = img.resize((vit_input_size, vit_input_size))
+
+            # Calculate the (row, column) position to place the frame within the grid layout.
+            local_idx = idx - start_idx
+            x_offset = (local_idx % max_grid_shape[0]) * vit_input_size
+            y_offset = (local_idx // max_grid_shape[0]) * vit_input_size
+
+            # Calculate the position to place the frame in the grid.
+            combined_image.paste(img_resized, (x_offset, y_offset))
+
+        # Append the current canvas to the result list.
+        image_list.append(combined_image)
+        frame_cnt = end_idx - start_idx
+        image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
+        time_stamp += frame_cnt * time_interval
+
+    if leftover_frames > 0:
+        # canvas_idx might be undefined; default to 0 if not previously assigned to avoid "referenced before assignment" error.
+        canvas_idx = num_canvases
+        # Add the remaining frames to the final canvas.
+        combined_image = Image.new("RGB", (vit_input_size * leftover_frames, vit_input_size * 1), color=(0, 0, 0))
+
+        for idx in range(leftover_frames):
+            img = frames[num_canvases * max_num_grids + idx]
+
+            # Resize the frame to a square (equal width and height).
+            img_resized = img.resize((vit_input_size, vit_input_size))
+
+            # Calculate the (row, column) position to place the frame within the grid layout.
+            x_offset = (idx % leftover_frames) * vit_input_size
+            y_offset = (idx // leftover_frames) * vit_input_size
+
+            # Calculate the position to place the frame within the grid layout.
+            combined_image.paste(img_resized, (x_offset, y_offset))
+
+        # Add the current canvas to the list of combined images.
+        image_list.append(combined_image)
+        frame_cnt = leftover_frames
+        image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
+        time_stamp += frame_cnt * time_interval
+
+    return image_list, image_time_stamps
+
+
+def extract_frame_indices(play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=0.4):
+    """
+    Extracts specific frame indices from a video based on duration, frame count, and sampling strategy.
+
+    The function determines which frames to extract given the video duration (`play_time`),
+    total frame count, and frame rate. It samples frames at regular intervals (default: 0.4s),
+    but if the number of frames exceeds the limit defined by `max_num_grids * max_image_cnt`,
+    it performs uniform sampling to stay within that limit.
+
+    Args:
+        play_time (float): Total play time of the video in seconds.
+        total_frames (int): Total number of frames in the video.
+        fps (float): Frames per second of the video.
+        max_num_grids (int): Maximum number of grids to display.
+        max_image_cnt (int): Maximum number of images per grid.
+        default_interval (float, optional): Interval in seconds between frame samples. Defaults to 0.4.
+
+    Returns:
+        Tuple:
+            frame_indices (List[int]): A list of selected frame indices.
+            time_interval (float): Time interval between selected frames (in seconds).
+    """
+
+    # Calculate how many frames to extract with the default interval
+    default_frame_count = int(play_time / default_interval)
+
+    # Maximum frames allowed based on max_num_grids and max_image_cnt
+    max_frames_allowed = max_num_grids * max_image_cnt
+
+    # Determine whether we can use the default interval or need uniform sampling
+    if default_frame_count <= max_frames_allowed:
+        # Default interval is sufficient, extract frames every 0.4 seconds
+        frame_interval = int(total_frames / default_frame_count)
+    else:
+        # Use uniform sampling to fit within max_frames_allowed
+        frame_interval = int(total_frames / max_frames_allowed)
+
+    # Extract frame indices at the calculated interval
+    selected_indices = list(range(0, total_frames, frame_interval))
+
+    time_interval = frame_interval / fps
+
+    # Ensure the number of selected indices does not exceed max_frames_allowed
+    return selected_indices[:max_frames_allowed], time_interval

preprocessor_config.json

@@ -0,0 +1,135 @@
+{
+  "processor_class": "HCXVisionProcessor",
+  "auto_map": {
+    "AutoProcessor": "preprocessor.HCXVisionProcessor"
+  },
+  "anyres": true,
+  "crop_size": {
+    "height": 378,
+    "width": 378
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "max_num_grids": 9,
+  "max_image_cnt": 12,
+  "num_queries_vis_abstractor": 81,
+  "num_queries_vis_abstractor_video_fast": 9,
+  "num_queries_vis_abstractor_video_slow": 81,
+  "first_last_frames_slow": false,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "HCXVisionProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pad_to_square": true,
+  "patch_size": 14,
+  "possible_resolutions": [
+    [
+      378,
+      378
+    ],
+    [
+      378,
+      756
+    ],
+    [
+      378,
+      1134
+    ],
+    [
+      378,
+      1512
+    ],
+    [
+      378,
+      1890
+    ],
+    [
+      378,
+      2268
+    ],
+    [
+      378,
+      2646
+    ],
+    [
+      378,
+      3024
+    ],
+    [
+      378,
+      3402
+    ],
+    [
+      756,
+      378
+    ],
+    [
+      756,
+      756
+    ],
+    [
+      756,
+      1134
+    ],
+    [
+      756,
+      1512
+    ],
+    [
+      1134,
+      378
+    ],
+    [
+      1134,
+      756
+    ],
+    [
+      1134,
+      1134
+    ],
+    [
+      1512,
+      378
+    ],
+    [
+      1512,
+      756
+    ],
+    [
+      1890,
+      378
+    ],
+    [
+      2268,
+      378
+    ],
+    [
+      2646,
+      378
+    ],
+    [
+      3024,
+      378
+    ],
+    [
+      3402,
+      378
+    ]
+  ],
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 378
+  },
+  "unpad": true
+}

pytorch_model-00001-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b34c808a1106a2d41f4a7202226d86d01a74b70fb267b2aa6a5e118b81806248
+size 1995404833

pytorch_model-00002-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c461a7f04797ab043c9a28dc90f8f60d5607e97a0a67b7d7660f760128d586df
+size 1963104270

pytorch_model-00003-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21b2e6e12f24b28beb82ddc329bc3177f6a8f3f105540801edb6c50cb7233276
+size 1988270158

pytorch_model-00004-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c50467661833482053fbbac1ea806cc818f09cde08545b731c7836667e91a99
+size 1495999777

pytorch_model.bin.index.json

@@ -0,0 +1,830 @@
+{
+  "metadata": {
+    "total_size": 7442487040
+  },
+  "weight_map": {
+    "image_newline": "pytorch_model-00001-of-00004.bin",
+    "language_model.lm_head.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.embed_tokens.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.10.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.10.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.11.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.12.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.13.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.13.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.13.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.13.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.13.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.13.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.13.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.13.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.13.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.14.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.14.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.15.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.16.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.17.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.18.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.19.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.2.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.2.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.2.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.2.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.2.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.2.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.2.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.2.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.2.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "language_model.model.layers.20.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.20.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.21.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.22.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.23.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.24.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.24.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.24.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.24.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "language_model.model.layers.25.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.25.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.26.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.27.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.28.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.29.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.3.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.3.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.30.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.30.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.input_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.mlp.down_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.mlp.gate_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.mlp.up_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.post_attention_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.self_attn.o_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.31.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "language_model.model.layers.4.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.4.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.5.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.6.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.7.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.8.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.layers.9.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "language_model.model.norm.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b1.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b2.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.0.b3.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b1.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b2.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv1.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv1.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv1.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv2.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv2.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv2.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv3.bn.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv3.bn.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.conv3.conv.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.se.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.se.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.se.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.net.2.b3.se.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.pos_emb": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.readout.0.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.readout.0.weight": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.readout.2.bias": "pytorch_model-00004-of-00004.bin",
+    "mm_projector.readout.2.weight": "pytorch_model-00004-of-00004.bin",
+    "vision_model.vision_model.embeddings.patch_embedding.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.embeddings.patch_embedding.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.embeddings.position_embedding.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.10.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.11.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.12.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.13.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.14.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.15.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.16.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.17.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.18.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.19.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.2.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.20.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.21.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.22.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.23.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.24.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.25.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.26.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.3.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.4.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.5.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.6.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.7.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.8.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.layer_norm1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.layer_norm1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.layer_norm2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.layer_norm2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.encoder.layers.9.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.attention.in_proj_bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.attention.in_proj_weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.attention.out_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.attention.out_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.layernorm.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.mlp.fc1.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.mlp.fc1.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.mlp.fc2.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.mlp.fc2.weight": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.head.probe": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.post_layernorm.bias": "pytorch_model-00001-of-00004.bin",
+    "vision_model.vision_model.post_layernorm.weight": "pytorch_model-00001-of-00004.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,80 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|endofprompt|>",
+    "<|_unuse_missing_100256|>",
+    "<|_unuse_missing_100261|>",
+    "<|_unuse_missing_100262|>",
+    "<|_unuse_missing_100263|>",
+    "<|_unuse_missing_100264|>",
+    "<|_unuse_missing_100265|>",
+    "<|_unuse_missing_100266|>",
+    "<|_unuse_missing_100267|>",
+    "<|_unuse_missing_100268|>",
+    "<|_unuse_missing_100269|>",
+    "<|_unuse_missing_100270|>",
+    "<|dummy3|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|stop|>",
+    "<|endofturn|>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": "<|endofturn|>",
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,507 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|_unuse_missing_100256|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100257": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100261": {
+      "content": "<|_unuse_missing_100261|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100262": {
+      "content": "<|_unuse_missing_100262|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100263": {
+      "content": "<|_unuse_missing_100263|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100264": {
+      "content": "<|_unuse_missing_100264|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|_unuse_missing_100265|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|_unuse_missing_100266|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100267": {
+      "content": "<|_unuse_missing_100267|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100268": {
+      "content": "<|_unuse_missing_100268|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100269": {
+      "content": "<|_unuse_missing_100269|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100270": {
+      "content": "<|_unuse_missing_100270|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100271": {
+      "content": "<|dummy3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100272": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100273": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100274": {
+      "content": "<|stop|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100275": {
+      "content": "<|endofturn|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100276": {
+      "content": "<|endofprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110491": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110492": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110493": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110494": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110495": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110496": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110497": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110498": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110499": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110500": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110501": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110502": {
+      "content": "<code_to_intermediate>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110503": {
+      "content": "<intermediate_to_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110504": {
+      "content": "<pr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110505": {
+      "content": "<pr_status>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110506": {
+      "content": "<pr_is_merged>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110507": {
+      "content": "<pr_base>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110508": {
+      "content": "<pr_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110509": {
+      "content": "<pr_base_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110510": {
+      "content": "<pr_diff>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110511": {
+      "content": "<pr_diff_hunk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110512": {
+      "content": "<pr_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110513": {
+      "content": "<pr_event_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110514": {
+      "content": "<pr_review>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110515": {
+      "content": "<pr_review_state>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110516": {
+      "content": "<pr_review_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110517": {
+      "content": "<pr_in_reply_to_review_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110518": {
+      "content": "<pr_in_reply_to_comment_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110519": {
+      "content": "<pr_diff_hunk_comment_line>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110520": {
+      "content": "<NAME>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110521": {
+      "content": "<EMAIL>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110522": {
+      "content": "<KEY>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110523": {
+      "content": "<PASSWORD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|endofprompt|>",
+    "<|_unuse_missing_100256|>",
+    "<|_unuse_missing_100261|>",
+    "<|_unuse_missing_100262|>",
+    "<|_unuse_missing_100263|>",
+    "<|_unuse_missing_100264|>",
+    "<|_unuse_missing_100265|>",
+    "<|_unuse_missing_100266|>",
+    "<|_unuse_missing_100267|>",
+    "<|_unuse_missing_100268|>",
+    "<|_unuse_missing_100269|>",
+    "<|_unuse_missing_100270|>",
+    "<|dummy3|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|stop|>",
+    "<|endofturn|>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "chat_template": [
+    {
+      "name": "default",
+      "template": "<|im_start|>tool_list\n<|im_end|>\n{% for message in messages %}\n{% set content = message['content'] %}\n{% set role = message['role'] %}\n{% if loop.first and role != 'system' %}\n<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}\n{% if message['content'] is string %}\n<|im_start|>{{ role }}\n{{ message['content'] }}<|im_end|>\n{% else %}\n{% if content['type'] == 'image' %}\n<|im_start|>{{ role }} (mime)\n{\"type\": \"image/jpeg\", \"filename\": \"{{ content['filename'] }}\"}<|im_end|>\n<|im_start|>{{ role }} (vector)\n<|dummy3|><|im_end|>\n<|im_start|>image/aux\n다음 중 ocr은 사진에서 검출된 글자이고, lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. 참고하여 답변하세요. {\"ocr\": \"{{ content['ocr'] or '' }}\", \"lens_keywords\": \"{{ content['lens_keywords'] or '' }}\", \"lens_local_keywords\": \"{{ content['lens_local_keywords'] or '' }}\"}<|im_end|>\n{% elif content['type'] == 'video' %}\n<|im_start|>{{ role }} (mime)\n{\"type\": \"video/mp4\", \"filename\": \"{{ content['filename'] }}\"}<|im_end|>\n<|im_start|>{{ role }} (vector)\n<|dummy3|><|im_end|>\n<|im_start|>image/aux\n{% if content.get('is_final_grid') %}\n다음 중 lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. speech_to_text는 비디오 속에서의 대화, 음성, 소리, 대사, 그리고 말을 전부 글로 받아 적은 것 입니다. 참고하여 답변하세요. {\"video_time_stamp\": \"{{ content['video_time_stamp'] }}\", \"lens_keywords\": \"{{ content.get('lens_keywords', '') }}\", \"lens_local_keywords\": \"{{ content.get('lens_local_keywords', '') }}\", \"speech_to_text\": \"{{ content.get('speech_to_text', '') }}\"}\n{% else %}\n다음 중 video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. 참고하여 답변하세요. {\"video_time_stamp\": \"{{ content['video_time_stamp'] }}\"}\n{% endif %}<|im_end|>\n{% elif content['type'] == 'text' %}\n<|im_start|>{{ role }}\n{{ content['text'] }}<|im_end|>\n{% endif %}\n{% endif %}\n{% endfor %}\n{% if add_generation_prompt %}\n<|im_start|>assistant\n{% endif %}\n"
+    }
+  ],
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endofturn|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}