update

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sample.jpg filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 SB Intuitions
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,121 @@
----
-license: mit
----
+---
+language:
+- ja
+- en
+base_model:
+- sbintuitions/sarashina2-7b
+license: mit
+tags:
+  - multimodal
+  - vision-language
+  - llama
+  - qwen2_vl
+pipeline_tag: image-to-text
+library_name: transformers
+---
+
+# Sarashina2-Vision-8B
+**Sarashina2-Vision-8B** is a Japanese Large Vision Language Model trained by [SB Intuitions](https://www.sbintuitions.co.jp/).
+
+This model is based on [Sarashina2-7B](https://huggingface.co/sbintuitions/sarashina2-7b) and Image Encoder of [Qwen2-VL-7B](https://huggingface.co/Qwen/Qwen2-VL-7B).
+
+It achieved the highest level of scores in 4 benchmarks (as of 2025/03/07) compared to other Japanese VLMs.
+
+## How to use
+### 1. Install dependencies
+
+```sh
+pip install -U transformers==4.47.0 torch torchvision pillow protobuf sentencepiece accelerate
+```
+
+### 2. Inference
+The following script loads the model and allows inference.
+```python
+import requests
+from PIL import Image
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+# Define model path
+model_path = "sbintuitions/sarashina2-vision-8b"
+
+# Load model and processor
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    device_map="cuda",
+    torch_dtype="auto",
+    trust_remote_code=True,
+)
+
+message = [{"role": "user", "content": "この写真に写っているもので、最も有名と考えられる建築物は何でどこに写っていますか？"}]
+text_prompt = processor.apply_chat_template(message, add_generation_prompt=True)
+"""text_prompt: <s><|prefix|><|file|><|suffix|>A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.
+
+### Human: この写真に写っているもので、最も有名と考えられる建築物は何でどこに写っていますか？
+### Assistant:"""
+
+sample_image_url = "https://huggingface.co/sbintuitions/sarashina2-vision-8b/resolve/main/sample.jpg"
+image = Image.open(requests.get(sample_image_url, stream=True).raw).convert("RGB")
+inputs = processor(
+    text=[text_prompt],
+    images=[image],
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+stopping_criteria = processor.get_stopping_criteria(["\n###"])
+
+# Inference: Generation of the output
+output_ids = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    temperature=0.0,
+    do_sample=False,
+    stopping_criteria=stopping_criteria,
+)
+generated_ids = [
+    output_ids[len(input_ids) :] for input_ids, output_ids in zip(inputs.input_ids, output_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
+)
+print(output_text[0])
+"""この写真に写っているもので、最も有名と考えられる建築物は東京タワーです。東京タワーは、東京のランドマークであり、この写真では、ビル群の向こうに写っています。"""
+```
+
+### Example
+<img src="https://huggingface.co/sbintuitions/sarashina2-vision-8b/resolve/main/sample.jpg" width="350">
+
+|Prompt|Output|
+|-|-|
+|この写真に写っているもので、最も有名と考えられる建築物は何でどこに写っていますか？|この写真に写っているもので、最も有名と考えられる建築物は東京タワーです。東京タワーは、東京のランドマークであり、この写真では、ビル群の向こうに写っています。|
+|真ん中に映っている赤と白の物は何ですか？|真ん中に映っている赤と白のものはクレーンです。|
+
+## Training
+**Sarashina2-Vision** is created through the following three-stage learning process:
+
+1. We tune the parameters in the projector by caption datasets.
+2. We tune the parameters in the Vision Encoder and projector by caption datasets.
+3. We tune the parameters in the projector and LLM by Visual Instruction datasets.
+
+## Evaluation Results
+|Model|Model Size|JMMMU<sup>*1</sup>|Heron-Bench<sup>*2</sup>|JDocQA|
+|-|-|-|-|-|
+|[heron-chat-git-ja-stablelm-base-7b-v1](https://huggingface.co/turing-motors/heron-chat-git-ja-stablelm-base-7b-v1)|7B|0.294|0.461|0.069|
+|[llava-calm2-siglip](https://huggingface.co/cyberagent/llava-calm2-siglip)|7B|0.07|0.521|0.084|
+|[Llama-3-EvoVLM-JP-v2](https://huggingface.co/SakanaAI/Llama-3-EvoVLM-JP-v2)|8B|0.389|0.509|0.103|
+|[Asagi-14B](https://huggingface.co/MIL-UT/Asagi-14B)|14B|0.302|0.433|0.06|
+|[llm-jp-3-vila-14b](https://huggingface.co/llm-jp/llm-jp-3-vila-14b)|14B|0.23|**0.665**|0.176|
+|[EZO-InternVL2-26B](https://huggingface.co/AXCXEPT/EZO-InternVL2-26B)|26B|0.389|0.609|0.196|
+|[Sarashina2-Vision-8B](https://huggingface.co/sbintuitions/sarashina2-vision-8b)|8B|0.393|0.648|0.229|
+|[Sarashina2-Vision-14B](https://huggingface.co/sbintuitions/sarashina2-vision-14b)|14B|**0.433**|0.644|**0.245**|
+
+1. Evaluated only single image samples (1,286 samples). If answer extraction failed, we treated it as incorrect (score 0) instead of making a random choice to eliminate stochasticity.
+2. GPT-4o (gpt-4o-2024-08-06) was used for LLM-as-a-Judge.
+
+
+## Ethical Considerations and Limitations
+Sarashina2-Vision might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs. Before using Sarashina2-Vision, we would like developers to tune models based on human preferences and safety considerations.
+
+## LICENSE
+[MIT License](./LICENSE)

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{{ bos_token + '<|prefix|><|file|><|suffix|>A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human\\'s questions.\\n\\n' }}{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Human: ' + message['content'] + '\\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'] + '\\n' }}{% endif %}{% endfor %}{% if messages[-1]['role'] == 'user' %}{{ '### Assistant:' }}{% endif %}"
+}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "Sarashina2VisionForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_sarashina2_vision.Sarashina2VisionConfig",
+    "AutoModelForCausalLM": "modeling_sarashina2_vision.Sarashina2VisionForCausalLM"
+  },
+  "end_image_token_index": 102398,
+  "image_token_index": 14,
+  "model_type": "sarashina2_vision",
+  "start_image_token_index": 102397,
+  "text_config": {
+    "_name_or_path": "sbintuitions/sarashina2-7b",
+    "architectures": [
+      "LlamaForCausalLM"
+    ],
+    "max_position_embeddings": 4096,
+    "model_type": "llama",
+    "rms_norm_eps": 1e-05,
+    "torch_dtype": "bfloat16",
+    "vocab_size": 102400
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.47.0",
+  "vision_config": {
+    "hidden_size": 4096,
+    "in_chans": 3,
+    "model_type": "qwen2_vl",
+    "spatial_patch_size": 14
+  }
+}

configuration_sarashina2_vision.py

@@ -0,0 +1,76 @@
+# coding=utf-8
+# Copyright 2025 the SB Intuitions.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Sarashina2Vision model configuration"""
+
+from typing import Any, Optional
+
+from transformers import LlamaConfig, PretrainedConfig
+from transformers.models.qwen2_vl.configuration_qwen2_vl import Qwen2VLVisionConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class Sarashina2VisionConfig(PretrainedConfig):
+    """
+    This is the configuration class to store the configuration of a [`Sarashina2VisionModel`]. It is used to instantiate a
+    Sarashina2Vision model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+    Args:
+        vision_config (`Dict`, *optional*):
+            The config for the visual encoder initialization.
+        text_config (`Dict`, *optional*):
+            The config for the text decoder initialization.
+        image_token_index (`int`):
+            image token id.
+        start_image_token_index (`int`):
+            start image token id.
+        end_image_token_index (`int`):
+            end image token id.
+    """
+
+    model_type = "sarashina2_vision"
+
+    def __init__(
+        self,
+        vision_config: Optional[dict[str, Any]] = None,
+        text_config: Optional[dict[str, Any]] = None,
+        image_token_index: int = 14,
+        start_image_token_index: int = 102397,
+        end_image_token_index: int = 102398,
+        **kwargs,
+    ):
+        if isinstance(text_config, dict):
+            self.text_config = LlamaConfig(**text_config)
+        elif isinstance(text_config, LlamaConfig):
+            self.text_config = text_config
+        elif text_config is None:
+            self.text_config = LlamaConfig()
+
+        if isinstance(vision_config, dict):
+            self.vision_config = Qwen2VLVisionConfig(**vision_config)
+        elif isinstance(vision_config, Qwen2VLVisionConfig):
+            self.vision_config = vision_config
+        elif vision_config is None:
+            self.vision_config = Qwen2VLVisionConfig()
+
+        self.image_token_index = image_token_index
+        self.start_image_token_index = start_image_token_index
+        self.end_image_token_index = end_image_token_index
+
+        super().__init__(**kwargs)

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.47.0"
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f848457d50b90f6e8d036d72892be4ff384af97d966dfe853ea7fefbc81d9b61
+size 9986932616

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5dc1e07621a0f4d8bcc96ec76492aa0ccc517f7b1a0cacdf6ad6c052bf6ea56
+size 6000187040

model.safetensors.index.json

@@ -0,0 +1,691 @@
+{
+  "metadata": {
+    "total_size": 15987043328
+  },
+  "weight_map": {
+    "llm.lm_head.weight": "model-00002-of-00002.safetensors",
+    "llm.model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "llm.model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.norm.weight": "model-00002-of-00002.safetensors",
+    "norm.bias": "model-00001-of-00002.safetensors",
+    "norm.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.0.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.1.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.10.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.11.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.12.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.13.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.14.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.15.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.16.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.17.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.18.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.19.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.2.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.20.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.21.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.22.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.23.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.24.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.25.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.26.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.27.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.28.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.29.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.3.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.30.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.31.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.4.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.5.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.6.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.7.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.8.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.attn.proj.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.attn.qkv.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.mlp.fc1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.mlp.fc2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.norm1.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.norm1.weight": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.norm2.bias": "model-00001-of-00002.safetensors",
+    "visual.blocks.9.norm2.weight": "model-00001-of-00002.safetensors",
+    "visual.merger.ln_q.bias": "model-00001-of-00002.safetensors",
+    "visual.merger.ln_q.weight": "model-00001-of-00002.safetensors",
+    "visual.merger.mlp.0.bias": "model-00001-of-00002.safetensors",
+    "visual.merger.mlp.0.weight": "model-00001-of-00002.safetensors",
+    "visual.merger.mlp.2.bias": "model-00001-of-00002.safetensors",
+    "visual.merger.mlp.2.weight": "model-00001-of-00002.safetensors",
+    "visual.patch_embed.proj.weight": "model-00001-of-00002.safetensors"
+  }
+}

modeling_sarashina2_vision.py

@@ -0,0 +1,242 @@
+# coding=utf-8
+# Copyright 2025 the SB Intuitions.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn import CrossEntropyLoss
+from transformers import (
+    AutoConfig,
+    AutoModelForCausalLM,
+    GenerationMixin,
+    LlamaForCausalLM,
+    PreTrainedModel,
+)
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.models.qwen2_vl.modeling_qwen2_vl import Qwen2VisionTransformerPretrainedModel
+from transformers.utils import logging, replace_return_docstrings
+
+from .configuration_sarashina2_vision import Sarashina2VisionConfig
+
+logger = logging.get_logger(__name__)
+
+_CONFIG_FOR_DOC = "Sarashina2VisionConfig"
+
+
+class Sarashina2VisionPreTrainedModel(PreTrainedModel):
+    config_class = Sarashina2VisionConfig
+    base_model_prefix = "model"
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+    _supports_cache_class = True
+    _supports_static_cache = True
+
+    def _init_weights(self, module):
+        std = (
+            self.config.initializer_range
+            if hasattr(self.config, "initializer_range")
+            else self.config.text_config.initializer_range
+        )
+
+        if hasattr(module, "class_embedding"):
+            module.class_embedding.data.normal_(mean=0.0, std=std)
+
+        if isinstance(module, (nn.Linear, nn.Conv3d)):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+class Sarashina2VisionForCausalLM(Sarashina2VisionPreTrainedModel, GenerationMixin):
+    def __init__(self, config: Sarashina2VisionConfig):
+        super().__init__(config)
+        self.visual = Qwen2VisionTransformerPretrainedModel._from_config(config.vision_config)
+        self.norm = nn.LayerNorm(config.text_config.hidden_size)
+        self.llm = LlamaForCausalLM._from_config(config.text_config)
+        self._attn_implementation = config._attn_implementation
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.llm.get_input_embeddings()
+
+    def get_image_embeds(
+        self,
+        hidden_states: torch.Tensor,
+        grid_thw: torch.Tensor,
+    ) -> torch.Tensor:
+        rotary_pos_emb = self.visual.rot_pos_emb(grid_thw)
+        hidden_states = self.visual.patch_embed(hidden_states)
+
+        cu_seqlens = torch.repeat_interleave(
+            grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]
+        ).cumsum(dim=0, dtype=torch.int32)
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
+
+        for blk in self.visual.blocks:
+            hidden_states = blk(
+                hidden_states, cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb
+            )
+        return self.norm(self.visual.merger(hidden_states))
+
+    @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        pixel_values: torch.FloatTensor = None,
+        image_grid_thw: Optional[torch.LongTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **lm_kwargs,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        """
+        Args:
+            input_ids (torch.LongTensor, optional): Indices of input sequence tokens in the vocabulary. Defaults to None.
+            attention_mask (Optional[torch.Tensor], optional):  Mask to avoid performing attention on padding token indices. Defaults to None.
+            position_ids (Optional[torch.LongTensor], optional): Indices of positions of each input sequence tokens in the position embeddings. Defaults to None.
+            past_key_values (Optional[List[torch.FloatTensor]], optional): _description_. Defaults to None.
+            inputs_embeds (Optional[torch.FloatTensor], optional): Instead of passing `input_ids` you can choose to directly pass an embedded representation. Defaults to None.
+            labels (Optional[torch.LongTensor], optional): Labels for computing the masked language modeling loss. Defaults to None.
+            use_cache (Optional[bool], optional): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding. Defaults to None.
+            output_attentions (Optional[bool], optional): Whether or not to return the attentions tensors of all attention layers. Defaults to None.
+            output_hidden_states (Optional[bool], optional): Whether or not to return the hidden states of all layers. Defaults to None.
+            return_dict (Optional[bool], optional): Whether or not to return a `CausalLMOutputWithPast` instead of a plain tuple. Defaults to None.
+            pixel_values (torch.FloatTensor, optional): The tensors corresponding to the input images. Defaults to None.
+            image_grid_thw (Optional[torch.LongTensor], optional): The temporal, height and width of feature shape of each image in LLM. Defaults to None.
+            cache_position (Optional[torch.LongTensor], optional): Indices depicting the position of the input sequence tokens in the sequence. Defaults to None.
+        Returns:
+            CausalLMOutputWithPast: The output of the model.
+        """
+        output_attentions = (
+            output_attentions if output_attentions is not None else self.config.output_attentions
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if inputs_embeds is None:
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+            if pixel_values is not None:
+                pixel_values = pixel_values.type(self.visual.get_dtype())
+                image_embeds = self.get_image_embeds(pixel_values, image_grid_thw)
+                n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
+                n_image_features = image_embeds.shape[0]
+                if n_image_tokens != n_image_features:
+                    raise ValueError(
+                        f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
+                    )
+                image_mask = (
+                    (input_ids == self.config.image_token_index)
+                    .unsqueeze(-1)
+                    .expand_as(inputs_embeds)
+                    .to(inputs_embeds.device)
+                )
+                image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
+                inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
+
+        outputs = self.llm(
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            cache_position=cache_position,
+            **lm_kwargs,
+        )
+
+        logits = outputs[0]
+
+        loss = None
+        if labels is not None:
+            # Upcast to float if we need to compute the loss to avoid potential precision issues
+            logits = logits.float()
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        inputs_embeds=None,
+        pixel_values=None,
+        attention_mask=None,
+        cache_position=None,
+        logits_to_keep=None,
+        image_grid_thw=None,
+        **kwargs,
+    ):
+        model_inputs = self.llm.prepare_inputs_for_generation(
+            input_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            logits_to_keep=logits_to_keep,
+            **kwargs,
+        )
+
+        if cache_position[0] == 0:
+            # If we're in cached decoding stage, pixel values should be None because input ids do not contain special image token anymore
+            # Otherwise we need pixel values to be passed to model
+            model_inputs["pixel_values"] = pixel_values
+            model_inputs["image_grid_thw"] = image_grid_thw
+
+        return model_inputs
+
+
+AutoConfig.register("sarashina2_vision", Sarashina2VisionConfig)
+AutoModelForCausalLM.register(Sarashina2VisionConfig, Sarashina2VisionForCausalLM)
+Sarashina2VisionConfig.register_for_auto_class()
+Sarashina2VisionForCausalLM.register_for_auto_class("AutoModelForCausalLM")

preprocessor_config.json

@@ -0,0 +1,32 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_sarashina2_vision.Srashina2VisionProcessor"
+  },
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Sarashina2VisionImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 1016064,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Srashina2VisionProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "max_pixels": 1016064,
+    "min_pixels": 3136
+  },
+  "temporal_patch_size": 2
+}

processing_sarashina2_vision.py

@@ -0,0 +1,383 @@
+# coding=utf-8
+# Copyright 2025 the SB Intuitions.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Processor class for Srashina2Vision.
+"""
+
+from copy import deepcopy
+from typing import List, Optional, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from PIL import Image
+from transformers import (
+    AutoImageProcessor,
+    PreTrainedTokenizer,
+    Qwen2VLImageProcessor,
+    StoppingCriteria,
+    StoppingCriteriaList,
+)
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_transforms import (
+    convert_to_rgb,
+    to_channel_dimension_format,
+)
+from transformers.image_utils import (
+    ChannelDimension,
+    ImageInput,
+    VideoInput,
+    get_image_size,
+    infer_channel_dimension_format,
+    is_scaled_image,
+    make_list_of_images,
+    to_numpy_array,
+)
+from transformers.models.qwen2_vl.image_processing_qwen2_vl import smart_resize
+from transformers.processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
+from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class GenerationStopper(StoppingCriteria):
+    def __init__(
+        self,
+        stop_str_list: list[str],
+        tokenizer: PreTrainedTokenizer,
+        decode_suffix_length: int = 5,
+    ):
+        self.stop_str_list = stop_str_list
+        self.tokenizer = deepcopy(tokenizer)
+        self.decode_suffix_length = decode_suffix_length
+        self.input_ids_end = None
+
+    def __repr__(self):
+        return f"Stopping words: {self.stop_str_list}"
+
+    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
+        if self.input_ids_end is None:
+            length = input_ids.shape[1]
+            self.input_ids_end = length - 1 if (length - 1) > 0 else 0
+        decode_ids = input_ids[0][self.input_ids_end :][-self.decode_suffix_length :]
+        if len(decode_ids) == 0:
+            decoded = ""
+        else:
+            decoded = self.tokenizer.decode(decode_ids)
+
+        for stop_str in self.stop_str_list:
+            if stop_str in decoded:
+                self.input_ids_end = None
+                return True
+        return False
+
+    @property
+    def criteria(self):
+        return StoppingCriteriaList([self])
+
+    def format(self, sentence: str):
+        for w in self.stop_str_list:
+            if w in sentence[-len(w) :]:
+                sentence = sentence[: -len(w)]
+        return sentence
+
+
+class Sarashina2VisionImageProcessor(Qwen2VLImageProcessor):
+    def _preprocess(
+        self,
+        images: Union[ImageInput, VideoInput],
+        do_resize: bool = None,
+        resample: Image.Resampling = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+    ):
+        """
+        Preprocess an image or batch of images. Copy of the `preprocess` method from `Qwen2VLImageProcessor`.
+
+        Args:
+            images (`ImageInput`):
+                Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`.
+            vision_info (`List[Dict]`, *optional*):
+                Optional list of dictionaries containing additional information about vision inputs.
+            do_resize (`bool`, *optional*, defaults to `self.do_resize`):
+                Whether to resize the image.
+            resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
+                Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums.
+            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
+                Whether to rescale the image.
+            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
+                Scale factor to use if rescaling the image.
+            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
+                Whether to normalize the image.
+            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
+                Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image.
+            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
+                Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image.
+            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
+                Whether to convert the image to RGB.
+            data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`):
+                The channel dimension format for the output image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - Unset: Use the channel dimension format of the input image.
+            input_data_format (`ChannelDimension` or `str`, *optional*):
+                The channel dimension format for the input image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.   - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
+        """
+        images = make_list_of_images(images)
+
+        if do_convert_rgb:
+            images = [convert_to_rgb(image) for image in images]
+
+        # All transformations expect numpy arrays.
+        images = [to_numpy_array(image) for image in images]
+
+        if do_rescale and is_scaled_image(images[0]):
+            logger.warning_once(
+                "It looks like you are trying to rescale already rescaled images. If the input"
+                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
+            )
+        if input_data_format is None:
+            # We assume that all images have the same channel dimension format.
+            input_data_format = infer_channel_dimension_format(images[0])
+
+        height, width = get_image_size(images[0], channel_dim=input_data_format)
+        resized_height, resized_width = height, width
+        processed_images = []
+        for image in images:
+            if do_rescale:
+                image = self.rescale(
+                    image, scale=rescale_factor, input_data_format=input_data_format
+                )
+
+            if do_normalize:
+                image = self.normalize(
+                    image=image,
+                    mean=image_mean,
+                    std=image_std,
+                    input_data_format=input_data_format,
+                )
+
+            image = to_channel_dimension_format(
+                image, data_format, input_channel_dim=input_data_format
+            )
+
+            if do_resize:
+                resized_height, resized_width = smart_resize(
+                    height,
+                    width,
+                    factor=self.patch_size * self.merge_size,
+                    min_pixels=self.min_pixels,
+                    max_pixels=self.max_pixels,
+                )
+                image = (
+                    F.interpolate(
+                        torch.from_numpy(image).unsqueeze(0),
+                        size=(resized_height, resized_width),
+                        mode="bicubic",
+                    )
+                    .squeeze(0)
+                    .numpy()
+                )
+
+            processed_images.append(image)
+
+        patches = np.array(processed_images)
+        if data_format == ChannelDimension.LAST:
+            patches = patches.transpose(0, 3, 1, 2)
+        if patches.shape[0] % self.temporal_patch_size != 0:
+            repeats = np.repeat(patches[-1][np.newaxis], self.temporal_patch_size - 1, axis=0)
+            patches = np.concatenate([patches, repeats], axis=0)
+        channel = patches.shape[1]
+        grid_t = patches.shape[0] // self.temporal_patch_size
+        grid_h, grid_w = resized_height // self.patch_size, resized_width // self.patch_size
+        patches = patches.reshape(
+            grid_t,
+            self.temporal_patch_size,
+            channel,
+            grid_h // self.merge_size,
+            self.merge_size,
+            self.patch_size,
+            grid_w // self.merge_size,
+            self.merge_size,
+            self.patch_size,
+        )
+        patches = patches.transpose(0, 3, 6, 4, 7, 2, 1, 5, 8)
+        flatten_patches = patches.reshape(
+            grid_t * grid_h * grid_w,
+            channel * self.temporal_patch_size * self.patch_size * self.patch_size,
+        )
+
+        return flatten_patches, (grid_t, grid_h, grid_w)
+
+
+class Srashina2VisionProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+        },
+    }
+
+
+class Srashina2VisionProcessor(ProcessorMixin):
+    r"""
+    Constructs Srashina2Vision processor which wraps a Srashina2Vision image processor and a LLama tokenizer into a single processor.
+    [`Srashina2VisionProcessor`] offers all the functionalities of [`Sarashina2VisionImageProcessor`] and [`LlamaTokenizerFast`]. See the
+    [`~Srashina2VisionProcessor.__call__`] and [`~Srashina2VisionProcessor.decode`] for more information.
+    Args:
+        image_processor ([`Sarashina2VisionImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`LlamaTokenizerFast`], *optional*):
+            The tokenizer is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer"]
+    valid_kwargs = ["chat_template"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = ("LlamaTokenizer", "LlamaTokenizerFast")
+
+    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
+        self.image_token = (
+            "<|file|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
+        )
+        self.stop_symbol = "\n###"
+        super().__init__(image_processor, tokenizer, chat_template=chat_template)
+
+    def __call__(
+        self,
+        images: ImageInput = None,
+        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
+        **kwargs: Unpack[Srashina2VisionProcessorKwargs],
+    ) -> BatchFeature:
+        """
+        Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text`
+        and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`] if `text` is not `None` to encode
+        the text. To prepare the vision inputs, this method forwards the `vision_infos` and `kwrags` arguments to
+        Sarashina2VisionImageProcessor's [`~Sarashina2VisionImageProcessor.__call__`] if `vision_infos` is not `None`.
+
+        Args:
+            images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`):
+                The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch
+                tensor. Both channels-first and channels-last formats are supported.
+            text (`str`, `List[str]`, `List[List[str]]`):
+                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
+                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
+                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
+            return_tensors (`str` or [`~utils.TensorType`], *optional*):
+                If set, will return tensors of a particular framework. Acceptable values are:
+                - `'tf'`: Return TensorFlow `tf.constant` objects.
+                - `'pt'`: Return PyTorch `torch.Tensor` objects.
+                - `'np'`: Return NumPy `np.ndarray` objects.
+                - `'jax'`: Return JAX `jnp.ndarray` objects.
+
+        Returns:
+            [`BatchFeature`]: A [`BatchFeature`] with the following fields:
+
+            - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`.
+            - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when
+              `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not
+              `None`).
+            - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`.
+            - **image_grid_thw** -- List of image 3D grid in LLM. Returned when `images` is not `None`.
+        """
+        output_kwargs = self._merge_kwargs(
+            Srashina2VisionProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        if images is not None:
+            image_inputs = self.image_processor(
+                images=images, videos=None, **output_kwargs["images_kwargs"]
+            )
+            image_grid_thw = image_inputs["image_grid_thw"]
+        else:
+            image_inputs = {}
+            image_grid_thw = None
+
+        if not isinstance(text, list):
+            text = [text]
+
+        if image_grid_thw is not None:
+            merge_length = self.image_processor.merge_size**2
+            index = 0
+            for i in range(len(text)):
+                while self.image_token in text[i]:
+                    text[i] = text[i].replace(
+                        self.image_token,
+                        "<|placeholder|>" * (image_grid_thw[index].prod() // merge_length),
+                        1,
+                    )
+                    index += 1
+                text[i] = text[i].replace("<|placeholder|>", self.image_token)
+
+        text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def batch_decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`].
+        """
+        return [
+            output.replace(self.stop_symbol, "")
+            for output in self.tokenizer.batch_decode(*args, **kwargs)
+        ]
+
+    def decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.decode`].
+        """
+        return self.tokenizer.decode(*args, **kwargs).replace(self.stop_symbol, "")
+
+    def post_process_image_text_to_text(self, generated_outputs):
+        """
+        Post-process the output of the model to decode the text.
+
+        Args:
+            generated_outputs (`torch.Tensor` or `np.ndarray`):
+                The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
+                or `(sequence_length,)`.
+
+        Returns:
+            `List[str]`: The decoded text.
+        """
+        return self.tokenizer.batch_decode(
+            generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False
+        )
+
+    @property
+    def model_input_names(self):
+        tokenizer_input_names = self.tokenizer.model_input_names
+        image_processor_input_names = self.image_processor.model_input_names
+        return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
+
+    def get_stopping_criteria(self, stop_symbols: List[str]):
+        stopping_criteria = GenerationStopper(stop_str_list=stop_symbols, tokenizer=self.tokenizer)
+        return stopping_criteria.criteria
+
+
+Srashina2VisionProcessor.register_for_auto_class("AutoProcessor")
+AutoImageProcessor.register("Sarashina2VisionImageProcessor", Sarashina2VisionImageProcessor)

processor_config.json

@@ -0,0 +1,6 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_sarashina2_vision.Srashina2VisionProcessor"
+  },
+  "processor_class": "Srashina2VisionProcessor"
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<cls>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<sep>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:008293028e1a9d9a1038d9b63d989a2319797dfeaa03f171093a57b33a3a8277
+size 1831879

tokenizer_config.json

@@ -0,0 +1,176 @@
+{
+  "add_bos_token": false,
+  "add_dummy_prefix_space": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<cls>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "<|available_tools|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "<|tool_calls|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "<|tool_results|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "<|code|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "<|file|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102397": {
+      "content": "<|prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102398": {
+      "content": "<|suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102399": {
+      "content": "<|middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "auto_map": {
+    "AutoProcessor": "processing_sarashina2_vision.Srashina2VisionProcessor"
+  },
+  "bos_token": "<s>",
+  "chat_template": "{{ bos_token + '<|prefix|><|file|><|suffix|>A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human\\'s questions.\\n\\n' }}{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Human: ' + message['content'] + '\\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'] + '\\n' }}{% endif %}{% endfor %}{% if messages[-1]['role'] == 'user' %}{{ '### Assistant:' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<cls>",
+  "do_lower_case": false,
+  "eos_token": "</s>",
+  "extra_ids": 0,
+  "extra_special_tokens": {},
+  "keep_accents": true,
+  "legacy": false,
+  "mask_token": "<mask>",
+  "model_max_length": 4096,
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Srashina2VisionProcessor",
+  "sep_token": "<sep>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}