Duplicate from naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B

Co-authored-by: HyperCLOVA X (admin) <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,62 @@
+HyperCLOVA X SEED Model License Agreement
+
+Model Release Date: April 24, 2025
+ 
+This HyperCLOVA X SEED Model License Agreement (the “Agreement”) is a legal agreement between you and NAVER Corporation and NAVER Cloud Corporation (“NAVER”) and governs your use of the Models that NAVER provides to You under this Agreement. 
+
+NAVER Corp., as the holder of the intellectual property of the Model, and its affiliate, NAVER Cloud Corp., as the exclusive business operator of HyperCLOVA X, enter into this Agreement with you. NAVER and you are each a “party” and collectively the “parties.”
+
+By using, reproducing, modifying, distributing, performing or displaying any portion or element of the Model or Derivative Model, or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement. You represent to us that you are lawfully able to enter into contracts, and if you are entering into this Agreement for an entity, that you have legal authority to bind that entity. 
+
+1.	Definitions. 
+
+1.1.	"Affiliate” means any entity directly or indirectly controlling, controlled by or under common control with either party, where “control” means the possession, directly or indirectly, of the power to independently direct or cause the direction of the management and policies of an entity, whether through ownership of more than fifty percent (50%) of the stock or other equity interests entitled to vote for representation on its board of directors, or body performing similar functions, by contract or otherwise.
+
+1.2.	“Derivative Model” means all (i) modifications to the Model, (ii) works based on the Model, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of the Model, to that model in order to cause that model to perform similarly to the Model, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by the Model for training that Model. For clarity, Outputs are not deemed Derivative Model. 
+
+1.3.	“Licensee” or “you” means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity’s behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
+
+1.4.	“Model” means the foundational large language models and software and algorithms, including machine-learning model code and trained model weights distributed by NAVER. 
+
+1.5.	“Output” means the information content output of the Model or a Derivative Model that results from operating or otherwise using the Model or Derivative Models. 
+
+2.	Conditions for Use, License Grant and Restrictions
+
+2.1.	Conditions for Use. The Model and any Derivative Model are subject to the terms of this Agreement and govern your use. If You institute copyright or patent litigation against any entity (including a crossclaim or counterclaim in a lawsuit) alleging that the Model or a Derivative Model constitutes direct or contributory copyright or patent infringement, then any license granted to you under this Agreement for that Model or Derivative Model will terminate as of the date such litigation is filed. NAVER may update this Agreement to comply with legal and regulatory requirements any time and You agree to either comply with any updated license or cease your copying, use, and distribution of the Model and any Derivative Model.
+
+2.2.	License Grant. Subject to the terms and conditions of this Agreement, NAVER hereby grants to you a non-exclusive, worldwide, non-transferable, revocable and royalty-free limited license under NAVER’s intellectual property or other rights owned by NAVER embodied in the Model to access, download, install, copy, use, reproduce, distribute, create derivative works of, and make modifications to the Model. 
+
+2.3.	Prohibited Use Policy. NAVER is committed to safety, trust and transparency in AI development. NAVER encourages You to (i) ensure that the product or service you develop, use, offer as a service or distributes meets the legal and ethical requirements of the relevant industry or use case, (ii) take reasonable measures to address unintended bias and to mitigate harm to others, including underrepresented or vulnerable groups, and (iii) inform users of the nature and limitations of the product or service. NAVER expressly prohibits the use of its products or services for any purpose in violation of applicable law and regulation, including but not limited to (a) illegal surveillance, (b) illegal collection or processing of biometric information without the consent of the subject where required under applicable law, or (c) illegal harassment, abuse, threatening or bullying of individuals or groups of individuals or intentionally misleading or deceiving others. 
+
+3.	Redistribution. 
+
+3.1.	You may reproduce, distribute or make available the Model or Derivative Models thereof, or a product or service (including another AI model) that contains any of them, if you meet all of the following conditions: you must (i) include the Prohibited Use Policy referenced in Section 2.3. as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of the Model or Derivative Model and you must provide notice to subsequence users you distribute to the Model or Derivative Models are subject to the use restrictions in Section 2.3., (ii) provide all third party recipients of the Model or Derivative Models a copy of this Agreement, (iii) cause any modified files to carry prominent notices stating that you modified the files; (iv) include the following attribution notice within a “Notice” text file distributed as part of such copies: “HyperCLOVA X SEED Model is licensed under the HyperCLOVA X SEED Model License Agreement, Copyright © NAVER Corp. All Rights Reserved.”, and (v) prominently display “Powered by HyperCLOVA X” on a related website, user interface, blogpost, about page, or product documentation. If you use the Model or any Outputs of the Model to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “HyperCLOVA X” at the beginning of any such AI model name.
+3.2.	You may add your own copyright statement to your modifications and, except as set forth in this Section, may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such Derivative Models as a whole, provided your use, reproduction, and distribution of the Model or Derivative Models otherwise comply with the terms and conditions stated in this Agreement. Any additional or different terms and conditions you impose must not conflict with the terms of this Agreement. 
+
+4.	Additional Commercial Terms. If (i) as of the Model Release Date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s Affiliates, is greater than 10 million monthly active users in the preceding calendar month, or (ii) the Licensee or its Affiliate distributes or makes available any product or service, which is substantially similar to or directly competes with any product and service provided by NAVER, then the Licensee must request a license from NAVER. Such license may be granted by NAVER at its sole discretion, and the Licensee is not authorized to exercise any rights under this Agreement unless and until NAVER expressly grants you such rights.
+
+5.	Generated Output. NAVER claims no rights in Outputs you generate using the Model. You and your use are solely responsible for Outputs and their subsequent uses. 
+
+6.	DISCLAIMER OF WARRANTY. UNLESS REQUIRED BY APPLICABLE LAW, THE MODEL AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR ANY KIND, AND NAVER DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MODEL, DERIVATIVE MODELS, OUTPUTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE MODEL AND ANY OUTPUTS AND RESULTS AND YOUR EXERCISE OF PERMISSION UNDER THIS AGREEMENT. 
+
+7.	LIMITATION OF LIABILITY. IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW (SUCH AS IN CASES OF DELIBERATE AND GROSSLY NEGLIGENT ACTS), WILL NAVER BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND, ARISING FROM OR RELATED TO THIS AGREEMENT, OR RESULTING FROMTHE USE OR INABILITY TO USE THE MODEL, DERIVATIVE MODELS OR, OUTPUTS (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGES, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES), EVEN IF NAVER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 
+
+8.	Indemnity. You will indemnify and hold harmless NAVER from and against any claim by any third party arising out of or related to your use or distribution of the Model, Derivative Model or Outputs. 
+
+9.	Intellectual Property.
+
+9.1.	This Agreement does not grant permission to use the trade names, trademarks, service marks, or product names of NAVER, except as required for reasonable and customary use in describing the origin of the Model and reproducing the content of the “Notice” text file. 
+
+9.2.	NAVER Corp. owns the Model and any Derivative Model created by NAVER Corp. Except as expressively granted in this Agreement, NAVER Corp. reserves all rights, interests and remedies in connection with the Model and Derivative Model created by NAVER Corp. and no other license or right is granted to you by implication, estoppel or otherwise. Subject to NAVER Corp.’s ownership of the Model and any Derivative Model made by or for NAVER Corp., with respect to any derivative works and modifications of the Model that are made by you, as between you and NAVER Corp., you are and will be the owner of such derivative works and modifications.
+
+10.	Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Model and will continue in full force and effect until terminated in accordance with the terms and conditions of this Agreement. NAVER may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Model and Derivative Model. Section 5, 6, 7 and 10 shall survive the termination of this Agreement. 
+
+11.	Governing Law and Jurisdiction. 
+
+11.1.	This Agreement will be governed by and construed in accordance with the laws of the Republic of Korea, without regard to its conflicts of laws principles. 
+
+11.2.	Any disputes, controversies, or claims arising out of or relating to this Agreement, including its existence, validity, interpretation, performance, breach, or termination, shall be referred to and finally resolved by arbitration administered by the Korean Commercial Arbitration Board (KCAB) in accordance with the International Arbitration Rules of the Korean Commercial Arbitration Board in force at the time of the commencement of the arbitration. The seat of arbitration shall be Seoul, Republic of Korea. The tribunal shall consist of one arbitrator. The language of the arbitration shall be English. Either party may seek interim or provisional relief from a court of competent jurisdiction, and doing so shall not be considered a waiver of any provision in this section. The arbitral tribunal also has the authority to issue orders for interim or provisional relief.
+
+12.	Modifications. NAVER reserves the right to modify or amend this Agreement at any time, in its sole discretion. Any modifications will be effective upon posting the updated Agreement on our website or through other means of communication. You are responsible for reviewing the Agreement periodically for changes. 
+
+13.	No Waiver. NAVER will not be treated as having waived any rights by not exercising (or delaying the exercise of) any rights under this Agreement.

README.md

@@ -0,0 +1,302 @@
+---
+license: other
+license_name: hyperclovax-seed
+license_link: LICENSE
+library_name: transformers
+---
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6512d9827fccffe1e9e28fa7/Lra7yfdthGdKcNk7vP5RS.png)
+
+
+## **Overview**
+
+HyperCLOVAX-SEED-Vision-Instruct-3B is a model developed by NAVER, built upon its proprietary backbone model and fine-tuned through post-training. It is capable of understanding both text and images, as well as generating text.
+
+The model is primarily designed with a focus on lightweight architecture, optimizing computational efficiency. In terms of visual understanding, it can handle visual question answering (VQA), chart and diagram interpretation, and even comprehend content. HyperCLOVAX-SEED-Vision-Instruct-3B aims for a Pareto-optimal balance specifically tuned for the Korean language, and it demonstrates competitive performance using fewer visual tokens compared to other models of similar size in inference scenarios.
+
+Particularly, the model shows relative strengths in handling Korean-language inputs and outperforms similarly sized open-source models in related benchmarks. As the first open-source vision-language model in Korea capable of visual understanding, it is expected to significantly contribute to strengthening Korea's sovereign AI capabilities.
+
+
+## **Updates**
+- **(2025.07.25)**: vLLM engine is available with [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed)
+- **(2025.07.08)**: Major code update for supporting vLLM engine ([link - related_discussion](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B/discussions/27))
+- **(2025.04.22)**: Initial release of the repository.
+
+
+## **Basic Information**
+
+- **Model Architecture**: LLaVA-based Vision-Language Model  
+    - **LLM Module**: Transformer-based architecture (Dense Model)  
+    - **Vision Encoder** : SigLIP-based architecture with 378x378px input resolution per grid.
+    - **Vision-Language Connector** : C-Abstractor based architecture with AnyRes mechanism, supporting up to 1.29M total pixels across 9 grids.
+- **Parameter Count**: 3.2B (LLM Module) + 0.43B (Vision Module)  
+- **Input/Output Format**: Text + Image + Video / Text  
+- **Context Length**: 16k  
+- **Knowledge Cutoff Date**: The model was trained on data collected before August 2024.  
+
+
+## **Training**
+
+#### **Text**
+
+Securing high-quality data is essential even during post-training, but having humans manually create or revise large-scale datasets posed significant limitations in terms of both cost and resources. Additionally, tasks requiring domain expertise were difficult to handle, and the risk of human error was high. To overcome these challenges, we utilized an automated validation system powered by HyperCLOVA X, which improved data quality and streamlined the training process — ultimately leading to enhanced overall model performance. As a result, the model showed significant improvements in areas with definitive answers, such as mathematics and coding.
+
+While reducing the cost of data collection is important, finding efficient training strategies is equally critical. HyperCLOVAX-SEED-Vision-Instruct-3B was developed starting from the HyperCLOVAX-SEED-Text-Base-3B and applied both Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) based on an online reinforcement algorithm called GRPO.
+
+#### **Vision**
+
+The Vision Understanding feature — where the model receives images and questions as input and generates text-based answers — was not part of the initial design of HyperCLOVA X. Therefore, the model architecture was carefully designed to add capabilities for handling vision-related tasks, such as image-based question answering (VQA) and chart/diagram interpretation, without compromising the existing performance of the HCX LLM. Special attention was given to handling auxiliary information within the input, especially considering the context length.
+
+Although HyperCLOVAX-SEED-Vision-Instruct-3B is a lightweight model, it is capable of performing basic image VQA tasks and even supports OCR-free processing. One of the key focus areas for this 3B model was optimizing the efficiency of video input tokens. Since input token length directly affects computational cost, the number of tokens extracted per frame was carefully adjusted to enable efficient video understanding with as few tokens as possible. Additionally, during the RLHF training phase, vision-specific V-RLHF data was used to enhance the model’s learning, just like in the text domain.
+
+## Benchmark
+#### Text
+
+| **Model** | **KMMLU (5-shot, acc)** | **HAE-RAE (5-shot, acc)** | **CLiCK (5-shot, acc)** | **KoBEST (5-shot, acc)** |
+|----------------------------|--------|---------|---------|-------|
+| HyperCLOVAX-SEED-Text-Base-3B  | 0.4847 | 0.7635  | 0.6386  | 0.7792 |
+| HyperCLOVAX-SEED-Vision-Instruct-3B| 0.4422 | 0.6499  | 0.5599  | 0.7180 |
+| Qwen2.5-3B-instruct        | 0.4451 | 0.6031  | 0.5649  | 0.7053 |
+| gemma-3-4b-it              | 0.3895 | 0.6059  | 0.5303  | 0.7262 |
+
+#### Vision
+
+| Model Name                        | Max Token Count per Video      | VideoMME (Ko) | NAVER-TV-CLIP (Ko) | VideoChatGPT (Ko) | PerceptionTest (En) | ActivityNet-QA (En) | KoNet (Ko) | MMBench-Val (En) | TextVQA-Val (En) | Korean VisIT-Bench (Ko) | Image (4 benchmarks) | Video (5 benchmarks) | All (9 benchmarks) |
+|-----------------------------------|--------------------------------|----------------|---------------------|--------------------|-----------------------|----------------------|------------|-------------------|-------------------|--------------------------|------------------------|------------------------|----------------------|
+| HyperCLOVAX-SEED-Vision-Instruct-3B              | 1856 tokens, 108 frames        | 48.2           | 61.0                | 53.6               | 55.2                  | 50.6                 | 69.2       | 81.8              | 79.2              | 37.0                     | 46.68                  | 53.70                  | 59.54                |
+| HyperCLOVAX-SEED-Vision-Instruct-3B (without OCR)| 1856 tokens, 108 frames        | 48.2           | 61.0                | 53.6               | 55.2                  | 50.6                 | 36.6       | 80.7              | 76.0              | 43.5                     | 56.74                  | 53.70                  | 55.05                |
+| Qwen-2.5-VL-3B                    | 24576 tokens, 768 frames       | 55.1           | 48.3                | 45.6               | 66.9                  | 55.7                 | 58.3       | 84.3              | 79.6              | 81.5                     | 59.35                  | 54.31                  | 56.55                |
+| Qwen-2.5-VL-3B (w/ 2000 tokens)   | 2000 tokens, 128 frames        | 50.3           | 43.9                | 44.3               | 58.3                  | 54.2                 | 58.5       | 84.3              | 79.3              | 15.7                     | 59.50                  | 50.18                  | 54.33                |
+| Qwen-2.5-VL-7B                    | 24576 tokens, 768 frames       | 60.6           | 66.7                | 51.8               | 70.5                  | 56.6                 | 68.4       | 88.3              | 84.9              | 85.6                     | 69.34                  | 61.23                  | 64.84                |
+| Gemma-3-4B                        | 4096 tokens, 16 frames         | 45.4           | 36.8                | 57.1               | 50.6                  | 46.3                 | 25.0       | 79.2              | 58.9              | 32.3                     | 48.91                  | 47.24                  | 47.98                |
+| GPT4V (gpt-4-turbo-2024-04-09)    | Unknown, Original Image , 8 frames | 49.1           | 75.0                | 55.5               | 57.4                  | 45.7                 | 38.7       | 84.2              | 60.4              | 52.0                     | 58.88                  | 51.59                  | 54.83                |
+| GPT4o (gpt-4o-2024-08-06)         | Unknown, 512 resize, 128 frames| 61.6           | 66.6                | 61.8               | 50.2                  | 41.7                 | 60.6       | 84.2              | 73.2              | 50.5                     | 67.15                  | 56.42                  | 61.19                |
+| InternV-2-2B                      | 4096 tokens, 16 frames         | 28.9           | 21.1                | 40.2               | 50.5                  | 50.3                 | 3.3        | 79.3              | 75.1              | 51.1                     | 39.74                  | 38.19                  | 38.88                |
+| InternV-2-4B                      | 4096 tokens, 16 frames         | 33.8           | 36.0                | 22.8               | 54.2                  | 52.0                 | 22.7       | 83.0              | 76.9              | 51.6                     | 46.11                  | 39.75                  | 42.58                |
+| InternV-2-8B                      | 4096 tokens, 16 frames         | 43.7           | 41.2                | 32.4               | 58.5                  | 53.2                 | 28.5       | 86.6              | 79.0              | 97.0                     | 50.32                  | 45.79                  | 47.81                |
+
+## Dependencies
+- [einops](https://einops.rocks/)
+- [timm](https://github.com/huggingface/pytorch-image-models)
+- [av](https://github.com/PyAV-Org/PyAV)
+- [decord](https://github.com/dmlc/decord)
+
+## Example
+**(code & benchmark score) checked with transformers 4.52.4**
+
+```python
+
+from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer
+
+model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
+model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(device="cuda")
+processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# LLM Example
+# It is recommended to use the chat template with HyperCLOVAX models.
+# Using the chat template allows you to easily format your input in ChatML style.
+llm_chat = [
+        {"role": "system", "content": [{"type": "text", "text": "you are helpful assistant!"}]},
+        {
+                "role": "user", 
+                "content": [
+                        {"type": "text", "text": "Hello, how are you?"},
+                        {"type": "text", "text": "I said. Hello, how are you today?"},
+                ]
+        },
+        {"role": "assistant", "content": [{"type": "text", "text": "I'm doing great. How can I help you today?"}]},
+        {"role": "user", "content": [{"type": "text", "text": "I'd like to show off how chat templating works!"}]},
+]
+model_inputs = processor.apply_chat_template(
+        llm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True
+)
+model_inputs = model_inputs.to(device="cuda")
+
+# Please adjust parameters like top_p appropriately for your use case.
+output_ids = model.generate(
+        **model_inputs,
+        max_new_tokens=64,
+        do_sample=True,
+        top_p=0.6,
+        temperature=0.5,
+        repetition_penalty=1.0,
+)
+print("=" * 80)
+print("LLM EXAMPLE")
+print(processor.batch_decode(output_ids)[0])
+print("=" * 80)
+
+# VLM Example
+# For images and videos, you can use url, local_path, base64, or bytes as input sources.
+vlm_chat = [
+        {"role": "system", "content": [{"text": "System Prompt", "type": "text"}]},
+        {"role": "user", "content": [{"text": "User Text Prompt 1", "type": "text"}]},
+        {
+                "role": "user",
+                "content": [{
+                        "filename": "tradeoff_sota.png",
+                        "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",
+                        "lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",
+                        "lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",
+                        "ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.",                        "type": "image",
+                }],
+        },
+        {
+                "role": "user",
+                "content": [{
+                        "filename": "tradeoff.png",
+                        "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",
+                        "type": "image",
+                }],
+        },
+        {"role": "assistant", "content": [{"text": "Assistant Text Prompt 1", "type": "text"}]},
+        {"role": "user", "content": [{"text": "User Text Prompt 2", "type": "text"}]},
+        {
+                "role": "user",
+                "content": [
+                        {
+                                "type": "video",
+                                "video": "freenaturestock-rolling-mist-clouds.mp4",
+                                "lens_keywords": "Prada re-edition, nylon bag, mini cross bag, logo strap, essential shoulder bag",
+                                "lens_local_keywords": "[0.12, 0.34, 0.85, 0.76] Prada re-edition",
+                                "speech_to_text": "Please enter the dialogue, voice, sound, lines, and words in the video in text format.",
+                        },
+                        {"text": "User Text Prompt 3", "type": "text"},
+                ]
+        },
+]
+
+model_inputs = processor.apply_chat_template(   
+        vlm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True,
+)
+model_inputs = model_inputs.to(device="cuda")
+output_ids = model.generate(
+        **model_inputs,
+        max_new_tokens=64,
+        do_sample=True,
+        top_p=0.6,
+        temperature=0.5,
+        repetition_penalty=1.0,
+)
+print("=" * 80)
+print("VLM EXAMPLE")
+print(processor.batch_decode(output_ids)[0])
+print("=" * 80)
+
+```
+
+## Example for v0.1.0
+**(code & benchmark score) checked with transformers 4.45.0**
+
+```python
+
+from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer
+
+model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
+revision="v0.1.0"
+model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, revision=revision).to(device="cuda")
+preprocessor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True, revision=revision)
+tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
+
+# LLM Example
+# It is recommended to use the chat template with HyperCLOVAX models.
+# Using the chat template allows you to easily format your input in ChatML style.
+chat = [
+        {"role": "system", "content": "you are helpful assistant!"},
+        {"role": "user", "content": "Hello, how are you?"},
+        {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
+        {"role": "user", "content": "I'd like to show off how chat templating works!"},
+]
+input_ids = tokenizer.apply_chat_template(chat, return_tensors="pt", tokenize=True)
+input_ids = input_ids.to(device="cuda")
+
+# Please adjust parameters like top_p appropriately for your use case.
+output_ids = model.generate(
+        input_ids,
+        max_new_tokens=64,
+        do_sample=True,
+        top_p=0.6,
+        temperature=0.5,
+        repetition_penalty=1.0,
+)
+print("=" * 80)
+print("LLM EXAMPLE")
+print(tokenizer.batch_decode(output_ids)[0])
+print("=" * 80)
+
+# VLM Example
+# For image and video inputs, you can use url, local_path, base64, or bytes.
+vlm_chat = [
+        {"role": "system", "content": {"type": "text", "text": "System Prompt"}},
+        {"role": "user", "content": {"type": "text", "text": "User Text 1"}},
+        {
+                "role": "user",
+                "content": {
+                        "type": "image",
+                        "filename": "tradeoff_sota.png",
+                        "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",
+                        "ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.",
+                        "lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",
+                        "lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",
+                }
+        },
+        {
+                "role": "user",
+                "content": {
+                        "type": "image",
+                        "filename": "tradeoff.png",
+                        "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",
+                }
+        },
+        {"role": "assistant", "content": {"type": "text", "text": "Assistant Text 1"}},
+        {"role": "user", "content": {"type": "text", "text": "User Text 2"}},
+        {
+                "role": "user",
+                "content": {
+                        "type": "video",
+                        "filename": "rolling-mist-clouds.mp4",
+                        "video": "freenaturestock-rolling-mist-clouds.mp4",
+                }
+        },
+        {"role": "user", "content": {"type": "text", "text": "User Text 3"}},
+]
+
+new_vlm_chat, all_images, is_video_list = preprocessor.load_images_videos(vlm_chat)
+preprocessed = preprocessor(all_images, is_video_list=is_video_list)
+input_ids = tokenizer.apply_chat_template(
+        new_vlm_chat, return_tensors="pt", tokenize=True, add_generation_prompt=True,
+)
+
+output_ids = model.generate(
+        input_ids=input_ids.to(device="cuda"),
+        max_new_tokens=8192,
+        do_sample=True,
+        top_p=0.6,
+        temperature=0.5,
+        repetition_penalty=1.0,
+        **preprocessed,
+)
+print("=" * 80)
+print("VLM EXAMPLE")
+print(tokenizer.batch_decode(output_ids)[0])
+print("=" * 80)
+```
+
+- To ensure the highest level of image understanding performance, it is recommended to include additional information such as Optical Character Recognition (OCR) results and entity recognition (Lens). The provided usage examples are written under the assumption that OCR and Lens results are available. If you input data in this format, you can expect significantly improved output quality.
+
+## vLLM
+To speed up your inference, you can use the vLLM engine from [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed).  
+
+Make sure to switch to the `v0.9.2rc2_hyperclovax_vision_seed` branch.  
+
+**Launch API server**:
+- https://oss.navercorp.com/HYPERSCALE-AI-VISION/vllm/blob/main/README.md
+
+**Request Example**:
+- https://github.com/vllm-project/vllm/pull/20931#issue-3229161410
+
+**Offline Inference Examples**:
+- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py
+- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py

added_tokens.json

@@ -0,0 +1,35 @@
+{
+  "<EMAIL>": 110521,
+  "<KEY>": 110522,
+  "<NAME>": 110520,
+  "<PASSWORD>": 110523,
+  "<code_to_intermediate>": 110502,
+  "<empty_output>": 110501,
+  "<file_sep>": 110492,
+  "<intermediate_to_code>": 110503,
+  "<issue_closed>": 110495,
+  "<issue_comment>": 110494,
+  "<issue_start>": 110493,
+  "<jupyter_code>": 110498,
+  "<jupyter_output>": 110499,
+  "<jupyter_script>": 110500,
+  "<jupyter_start>": 110496,
+  "<jupyter_text>": 110497,
+  "<pr>": 110504,
+  "<pr_base>": 110507,
+  "<pr_base_code>": 110509,
+  "<pr_comment>": 110512,
+  "<pr_diff>": 110510,
+  "<pr_diff_hunk>": 110511,
+  "<pr_diff_hunk_comment_line>": 110519,
+  "<pr_event_id>": 110513,
+  "<pr_file>": 110508,
+  "<pr_in_reply_to_comment_id>": 110518,
+  "<pr_in_reply_to_review_id>": 110517,
+  "<pr_is_merged>": 110506,
+  "<pr_review>": 110514,
+  "<pr_review_comment>": 110516,
+  "<pr_review_state>": 110515,
+  "<pr_status>": 110505,
+  "<repo_name>": 110491
+}

chat_template.jinja

@@ -0,0 +1,65 @@
+<|im_start|>tool_list
+<|im_end|>
+{% for message in messages %}
+{% set content = message['content'] %}
+{% set role = message['role'] %}
+{% if loop.first and role != 'system' %}
+<|im_start|>system
+You are a helpful assistant.<|im_end|>
+{% endif %}
+{% if message['content'] is string %}
+<|im_start|>{{ role }}
+{{ message['content'] }}<|im_end|>
+{% elif message['content'] is mapping %}
+{% if content['type'] == 'image' %}
+<|im_start|>{{ role }} (mime)
+{"type": "image/jpeg", "filename": "{{ content['filename'] }}"}<|im_end|>
+<|im_start|>{{ role }} (vector)
+<|dummy3|><|im_end|>
+<|im_start|>image/aux
+다음 중 ocr은 사진에서 검출된 글자이고, lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. 참고하여 답변하세요. {"ocr": "{{ content['ocr'] or '' }}", "lens_keywords": "{{ content['lens_keywords'] or '' }}", "lens_local_keywords": "{{ content['lens_local_keywords'] or '' }}"}<|im_end|>
+{% elif content['type'] == 'video' %}
+<|im_start|>{{ role }} (mime)
+{"type": "video/mp4", "filename": "{{ content['filename'] }}"}<|im_end|>
+<|im_start|>{{ role }} (vector)
+<|_unuse_missing_100270|><|im_end|>
+<|im_start|>image/aux
+{% if content.get('is_final_grid') %}
+다음 중 lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. speech_to_text는 비디오 속에서의 대화, 음성, 소리, 대사, 그리고 말을 전부 글로 받아 적은 것 입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}", "lens_keywords": "{{ content.get('lens_keywords', '') }}", "lens_local_keywords": "{{ content.get('lens_local_keywords', '') }}", "speech_to_text": "{{ content.get('speech_to_text', '') }}"}
+{% else %}
+다음 중 video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}"}
+{% endif %}<|im_end|>
+{% elif content['type'] == 'text' %}
+<|im_start|>{{ role }}
+{{ content['text'] }}<|im_end|>
+{% endif %}
+{% elif message['content'] is sequence %}
+{% for content in message['content'] %}
+{% if content['type'] == 'image' %}
+<|im_start|>{{ role }} (mime)
+{"type": "image/jpeg", "filename": "{{ content['filename'] }}"}<|im_end|>
+<|im_start|>{{ role }} (vector)
+<|dummy3|><|im_end|>
+<|im_start|>image/aux
+다음 중 ocr은 사진에서 검출된 글자이고, lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. 참고하여 답변하세요. {"ocr": "{{ content['ocr'] or '' }}", "lens_keywords": "{{ content['lens_keywords'] or '' }}", "lens_local_keywords": "{{ content['lens_local_keywords'] or '' }}"}<|im_end|>
+{% elif content['type'] == 'video' %}
+<|im_start|>{{ role }} (mime)
+{"type": "video/mp4", "filename": "{{ content['filename'] }}"}<|im_end|>
+<|im_start|>{{ role }} (vector)
+<|_unuse_missing_100270|><|im_end|>
+<|im_start|>image/aux
+{% if content.get('is_final_grid') %}
+다음 중 lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. speech_to_text는 비디오 속에서의 대화, 음성, 소리, 대사, 그리고 말을 전부 글로 받아 적은 것 입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}", "lens_keywords": "{{ content.get('lens_keywords', '') }}", "lens_local_keywords": "{{ content.get('lens_local_keywords', '') }}", "speech_to_text": "{{ content.get('speech_to_text', '') }}"}
+{% else %}
+다음 중 video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}"}
+{% endif %}<|im_end|>
+{% elif content['type'] == 'text' %}
+<|im_start|>{{ role }}
+{{ content['text'] }}<|im_end|>
+{% endif %}
+{% endfor %}
+{% endif %}
+{% endfor %}
+{% if add_generation_prompt %}
+<|im_start|>assistant
+{% endif %}

config.json

@@ -0,0 +1,202 @@
+{
+  "anyres": true,
+  "architectures": [
+    "HCXVisionForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_hyperclovax.HCXVisionConfig",
+    "AutoModelForCausalLM": "modeling_hyperclovax.HCXVisionForCausalLM"
+  },
+  "decoder_max_length": 16384,
+  "freeze_decoder": false,
+  "freeze_encoder": true,
+  "freeze_mm_projector": false,
+  "hidden_size": 3072,
+  "ignore_index": -100,
+  "video_token_id": 100270,
+  "image_token_id": 100271,
+  "mm_projector_type": "cabstractor",
+  "text_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "LlamaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 100257,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "end_token_id": 100257,
+    "eos_token_id": 100257,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 3072,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 7168,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "logits_scaling": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 131072,
+    "min_length": 0,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 24,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 100257,
+    "prefix": null,
+    "pretraining_tp": 1,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "resid_pdrop": 0.2,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 100000000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "transformers_version": "4.52.4",
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 110592
+  },
+  "max_image_cnt": 12,
+  "max_num_grids": 9,
+  "model_type": "hyperclovax_vlm",
+  "num_queries_vis_abstractor_image": 81,
+  "num_queries_vis_abstractor_video_slow": 81,
+  "num_queries_vis_abstractor_video_fast": 9,
+  "first_last_frames_slow": false,
+  "proj_pos_emb": true,
+  "proj_prenorm": false,
+  "q_former_model_name_or_path": null,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.52.4",
+  "unpad": true,
+  "use_1x1_grid": true,
+  "use_nth_layer": -2,
+  "vision_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "SiglipVisionModel"
+    ],
+    "attention_dropout": 0.0,
+    "auto_map": {},
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 378,
+    "initializer_factor": 1.0,
+    "intermediate_size": 4304,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_num_grids": 9,
+    "min_length": 0,
+    "model_type": "siglip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 14,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "transformers_version": "4.52.4",
+    "typical_p": 1.0,
+    "use_bfloat16": true
+  }
+}

configuration_hyperclovax.py

@@ -0,0 +1,66 @@
+from transformers import AutoConfig
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class HCXVisionConfig(PretrainedConfig):
+    model_type = "hyperclovax_vlm"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # The `gpt2` class has a different name, so it needs to be updated accordingly.
+    text_config_attribute_map = {
+        "n_embd": "hidden_size",
+        "n_positions": "max_position_embeddings",
+        "n_head": "num_attention_heads",
+        "n_layer": "num_hidden_layers",
+    }
+
+    def __init__(
+        self,
+        text_config=None,
+        vision_config=None,
+        use_nth_layer=-2,
+        img_start_id=100009,  # <|dummy3|>
+        decoder_max_length=4096,
+        anyres=False,
+        unpad=False,
+        max_num_grids=-1,
+        num_queries_vis_abstractor=-1,
+        ignore_index=-100,
+        proj_pos_emb=True,
+        proj_prenorm=False,
+        use_1x1_grid=False,
+        **kwargs,
+    ):
+        for key, val in self.text_config_attribute_map.items():
+            if text_config is not None and key in text_config:
+                text_config[val] = text_config.pop(key)
+
+        if text_config is not None:
+            _text_config = AutoConfig.for_model(text_config["model_type"])
+            self.text_config = _text_config.from_dict(text_config)
+
+            # In DeepSpeed ZeRO-3, the memory size is automatically determined based on the `hidden_size` specified in the config.
+            self.hidden_size = text_config["hidden_size"] if "hidden_size" in text_config else text_config["n_embd"]
+        if vision_config is not None:
+            _vision_config = AutoConfig.for_model(vision_config["model_type"])
+            self.vision_config = _vision_config.from_dict(vision_config)
+
+        # add VLM configs
+        self.use_nth_layer = use_nth_layer
+        self.decoder_max_length = decoder_max_length
+        self.anyres = anyres
+        self.unpad = unpad
+        self.max_num_grids = max_num_grids
+        self.num_queries_vis_abstractor = num_queries_vis_abstractor
+        self.img_start_id = img_start_id
+        self.ignore_index = ignore_index
+        self.proj_pos_emb = proj_pos_emb
+        self.proj_prenorm = proj_prenorm
+        self.use_1x1_grid = use_1x1_grid
+        super().__init__(**kwargs)
+
+    def get_text_config(self, decoder=False):
+        return self.text_config

image_processing_hyperclovax.py

@@ -0,0 +1,789 @@
+import copy
+import math
+import os
+from typing import Dict, List, Optional, Union
+
+import numpy as np
+import torch
+from PIL import Image
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_processing_utils import (
+    BaseImageProcessor,
+    get_size_dict,
+)
+from transformers.image_transforms import (
+    convert_to_rgb,
+    get_resize_output_image_size,
+    resize,
+    to_channel_dimension_format,
+)
+from transformers.image_utils import (
+    OPENAI_CLIP_MEAN,
+    OPENAI_CLIP_STD,
+    ChannelDimension,
+    ImageInput,
+    PILImageResampling,
+    get_image_size,
+    infer_channel_dimension_format,
+    is_scaled_image,
+    make_list_of_images,
+    to_numpy_array,
+    valid_images,
+)
+from transformers.utils import TensorType, logging
+
+logger = logging.get_logger(__name__)
+
+
+class HCXImageProcessor(BaseImageProcessor):
+    r"""
+    Constructs a VLM image processor. Based on [`CLIPImageProcessor`] with incorporation of additional techniques for processing high resolution images.
+    Args:
+        anyres: (bool) anyres 기능을 사용할지 안할지
+        unpad: (bool) anyres 사용시, unpad 기능 (순수 pad 영역에 해당하는 visual tokens 은 LLM input 에서 제거) 을 사용할지 안할지
+        num_queries_vis_abstractor: (int) 각 grid 에 대해서 resampler 를 사용하는 경우, visual query 수
+        possible_resolutions: (List) anyres 기능 사용시, 가능한 resolution 조합, 예: [[336, 336], [336, 672], [672, 336]]
+        patch_size: (int) ViT patch size
+        pad_to_square: (bool) 정사각형으로 padding 을 수행할지, 안할지를 결정. False 이면 정사각형이 아니기 때문에 center crop 을 거쳐 ViT 의 입력으로 들어감
+    """
+
+    model_input_names = ["pixel_values"]
+
+    def __init__(
+        self,
+        do_resize: bool = True,
+        size: Dict[str, int] = None,
+        anyres: bool = False,
+        unpad: bool = False,
+        num_queries_vis_abstractor_image: int = 81,
+        num_queries_vis_abstractor_video_slow: int = 81,
+        num_queries_vis_abstractor_video_fast: int = 9,
+        first_last_frames_slow_video: bool = False,
+        possible_resolutions: List = [],
+        patch_size: int = 14,
+        pad_to_square: bool = True,
+        resample: PILImageResampling = PILImageResampling.BICUBIC,
+        do_center_crop: bool = True,
+        crop_size: Dict[str, int] = None,
+        do_rescale: bool = True,
+        rescale_factor: Union[int, float] = 1 / 255,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = True,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        size = size if size is not None else {"shortest_edge": 336}
+        size = get_size_dict(size, default_to_square=False)
+        crop_size = crop_size if crop_size is not None else {"height": 336, "width": 336}
+        crop_size = get_size_dict(crop_size, default_to_square=True, param_name="crop_size")
+
+        self.do_resize = do_resize
+        self.size = size
+        self.anyres = anyres
+        self.unpad = unpad
+        self.num_queries_vis_abstractor_image = num_queries_vis_abstractor_image
+        self.num_queries_vis_abstractor_video_slow = num_queries_vis_abstractor_video_slow
+        self.num_queries_vis_abstractor_video_fast = num_queries_vis_abstractor_video_fast
+        self.first_last_frames_slow_video = first_last_frames_slow_video
+        self.possible_resolutions = [_resolution for _resolution in possible_resolutions]
+        self.patch_size = patch_size
+        self.pad_to_square = pad_to_square
+        self.resample = resample
+        self.do_center_crop = do_center_crop
+        self.crop_size = crop_size
+        self.do_rescale = do_rescale
+        self.rescale_factor = rescale_factor
+        self.do_normalize = do_normalize
+        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
+        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
+        self.do_convert_rgb = do_convert_rgb
+
+    def resize(
+        self,
+        image: np.ndarray,
+        size: Dict[str, int],
+        resample: PILImageResampling = PILImageResampling.BICUBIC,
+        data_format: Optional[Union[str, ChannelDimension]] = None,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+        **kwargs,
+    ) -> np.ndarray:
+        default_to_square = True
+        if "shortest_edge" in size:
+            size = size["shortest_edge"]
+            default_to_square = False
+        elif "height" in size and "width" in size:
+            size = (size["height"], size["width"])
+        else:
+            raise ValueError("Size must contain either 'shortest_edge' or 'height' and 'width'.")
+
+        output_size = get_resize_output_image_size(
+            image,
+            size=size,
+            default_to_square=default_to_square,
+            input_data_format=input_data_format,
+        )
+
+        return resize(
+            image,
+            size=output_size,
+            resample=resample,
+            data_format=data_format,
+            input_data_format=input_data_format,
+            **kwargs,
+        )
+
+    def _preprocess(
+        self,
+        images: ImageInput,
+        do_resize: bool = None,
+        size: Dict[str, int] = None,
+        resample: PILImageResampling = None,
+        do_center_crop: bool = None,
+        crop_size: int = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+    ) -> Image.Image:
+        images = make_list_of_images(images)
+
+        if do_resize:
+            images = [
+                self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)
+                for image in images
+            ]
+
+        if do_center_crop:
+            images = [
+                self.center_crop(image=image, size=crop_size, input_data_format=input_data_format) for image in images
+            ]
+
+        if do_rescale:
+            images = [
+                self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format) for image in images
+            ]
+
+        if do_normalize:
+            images = [
+                self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
+                for image in images
+            ]
+
+        images = [
+            to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
+        ]
+
+        return images
+
+    def _resize_for_local_grids(
+        self, image: np.array, target_resolution: tuple, resample, input_data_format: ChannelDimension
+    ) -> np.array:
+        new_height, new_width = _get_local_grids_output_size(image, target_resolution, input_data_format)
+
+        # Resize the image
+        resized_image = resize(image, (new_height, new_width), resample=resample, input_data_format=input_data_format)
+
+        return resized_image
+
+    def _pad_for_patching(
+        self, image: np.array, target_resolution: tuple, input_data_format: ChannelDimension
+    ) -> np.array:
+        """
+        Pad an image to a target resolution while maintaining aspect ratio.
+        """
+        target_height, target_width = target_resolution
+
+        background_color = tuple(int(x * 255) for x in self.image_mean)
+        padded_image = pad(
+            image,
+            target_size=(target_height, target_width),
+            background_color=background_color,
+            input_data_format=input_data_format,
+        )
+
+        return padded_image
+
+    def get_image_grids(
+        self,
+        image: np.array,
+        possible_resolutions,
+        grid_size: int,
+        resample: PILImageResampling,
+        data_format: ChannelDimension,
+        input_data_format: ChannelDimension,
+    ) -> List[np.array]:
+        if not isinstance(possible_resolutions, list):
+            raise ValueError("possible_resolutions must be a list of possible resolutions.")
+
+        image_size = get_image_size(image, channel_dim=input_data_format)
+        best_resolution = select_best_resolution(image_size, possible_resolutions)
+        resized_image = self._resize_for_local_grids(
+            image, best_resolution, resample=resample, input_data_format=input_data_format
+        )
+        padded_image = self._pad_for_patching(resized_image, best_resolution, input_data_format=input_data_format)
+        local_grids = divide_to_grids(padded_image, grid_size=grid_size, input_data_format=input_data_format)
+
+        # make sure that all patches are in the input data format
+        local_grids = [
+            to_channel_dimension_format(grid, channel_dim=data_format, input_channel_dim=input_data_format)
+            for grid in local_grids
+        ]
+
+        return local_grids
+
+    def preprocess(
+        self,
+        images: ImageInput,
+        do_resize: bool = None,
+        size: Dict[str, int] = None,
+        anyres: bool = None,
+        unpad: bool = None,
+        is_video: bool = False,
+        num_queries_vis_abstractor_image: int = None,
+        num_queries_vis_abstractor_video_slow: int = None,
+        num_queries_vis_abstractor_video_fast: int = None,
+        first_last_frames_slow_video: bool = None,
+        possible_resolutions: List = None,
+        patch_size: int = None,
+        pad_to_square: bool = None,
+        resample: PILImageResampling = None,
+        do_center_crop: bool = None,
+        crop_size: int = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+        return_dummy_image: bool = False,
+        first_last_frames_slow: bool = False,
+        is_first_or_last_frames: bool = False,
+        **kwargs,
+    ):
+        """
+        HCXVisionImageProcessor 로 image tensor, original image size (width, height), visual tokens
+        :return pixel_values: List of 4D tensor 로 image tensor
+        :return image_sizes: List of Dict 로 image width, height [{"width": image 1 의 width, "height": image 1 의 height}, {"width": image 2 의 width, "height": image 2 의 height}, ...]
+        :return vision_query_lengths: List of int 로 각 image 가 LLM 입력으로 전달될때 변환되는 visual token 수
+        """
+
+        do_resize = do_resize if do_resize is not None else self.do_resize
+        size = size if size is not None else self.size
+        size = get_size_dict(size, param_name="size", default_to_square=False)
+        anyres = anyres if anyres is not None else self.anyres
+        unpad = unpad if unpad is not None else self.unpad
+        num_queries_vis_abstractor_image = (
+            num_queries_vis_abstractor_image
+            if num_queries_vis_abstractor_image is not None
+            else self.num_queries_vis_abstractor_image
+        )
+        num_queries_vis_abstractor_video_slow = (
+            num_queries_vis_abstractor_video_slow
+            if num_queries_vis_abstractor_video_slow is not None
+            else self.num_queries_vis_abstractor_video_slow
+        )
+        num_queries_vis_abstractor_video_fast = (
+            num_queries_vis_abstractor_video_fast
+            if num_queries_vis_abstractor_video_fast is not None
+            else self.num_queries_vis_abstractor_video_fast
+        )
+        first_last_frames_slow_video = (
+            first_last_frames_slow_video
+            if first_last_frames_slow_video is not None
+            else self.first_last_frames_slow_video
+        )
+        possible_resolutions = possible_resolutions if possible_resolutions is not None else self.possible_resolutions
+        patch_size = patch_size if patch_size is not None else self.patch_size
+        pad_to_square = pad_to_square if pad_to_square is not None else self.pad_to_square
+        resample = resample if resample is not None else self.resample
+        do_center_crop = do_center_crop if do_center_crop is not None else self.do_center_crop
+        crop_size = crop_size if crop_size is not None else self.crop_size
+        crop_size = get_size_dict(crop_size, param_name="crop_size", default_to_square=True)
+        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
+        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
+        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
+        image_mean = image_mean if image_mean is not None else self.image_mean
+        image_std = image_std if image_std is not None else self.image_std
+        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
+
+        if is_video:
+            num_queries_vis_abstractor = num_queries_vis_abstractor_video_fast
+            num_queries_vis_abstractor_slow = num_queries_vis_abstractor_video_slow
+            unpad = False
+        else:
+            num_queries_vis_abstractor = num_queries_vis_abstractor_image
+            num_queries_vis_abstractor_slow = 0
+
+        if return_dummy_image:
+            images = Image.new("RGB", (224, 224), (0, 0, 0))
+
+        images = make_list_of_images(images)
+
+        if not valid_images(images):
+            raise ValueError(
+                "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
+                "torch.Tensor, tf.Tensor or jax.ndarray."
+            )
+
+        if do_convert_rgb:
+            images = [convert_to_rgb(image) for image in images]
+
+        # All transformations expect numpy arrays.
+        images = [to_numpy_array(image) for image in images]
+
+        if is_scaled_image(images[0]) and do_rescale:
+            logger.warning_once(
+                "It looks like you are trying to rescale already rescaled images. If the input"
+                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
+            )
+
+        if input_data_format is None:
+            # We assume that all images have the same channel dimension format.
+            input_data_format = infer_channel_dimension_format(images[0])
+
+        new_images = []
+        image_sizes = [get_image_size(image, channel_dim=input_data_format) for image in images]
+        vision_query_lengths = []
+
+        assert crop_size["height"] == crop_size["width"]
+
+        # global image 의 padding 연산은, image original width, height 가 클 때 bottleneck 이 될 수 있음
+        # 장축의 길이를 size["shortest_edge"] 로 resize 를 먼저 한 뒤에, padding
+        if anyres:
+            anyres_global_images = copy.deepcopy(images)
+            if pad_to_square:
+                background_color = tuple(int(x * 255) for x in self.image_mean)
+                anyres_global_images = [
+                    resize_longside(copy.deepcopy(image), size["shortest_edge"], resample, input_data_format)
+                    for image in anyres_global_images
+                ]
+                anyres_global_images = [
+                    expand2square(image, background_color=background_color, input_data_format=input_data_format)[0]
+                    for image in anyres_global_images
+                ]
+            else:
+                anyres_global_images = [
+                    self.resize(
+                        image=image,
+                        size={"height": size["shortest_edge"], "width": size["shortest_edge"]},
+                        resample=resample,
+                        input_data_format=input_data_format,
+                    )
+                    for image in anyres_global_images
+                ]
+        else:
+            anyres_global_images = [None for _ in range(len(images))]
+            if pad_to_square:
+                background_color = tuple(int(x * 255) for x in self.image_mean)
+                images = [
+                    resize_longside(image, size["shortest_edge"], resample, input_data_format) for image in images
+                ]
+                images = [
+                    expand2square(image, background_color=background_color, input_data_format=input_data_format)[0]
+                    for image in images
+                ]
+
+        for image, anyres_global_image, image_size in zip(images, anyres_global_images, image_sizes):
+            if anyres:
+                # convert image into a list of grids
+                # we intentially use the same data format as the input data format
+                image_grids = self.get_image_grids(
+                    image,
+                    possible_resolutions,
+                    grid_size=crop_size["height"],
+                    resample=resample,
+                    data_format=input_data_format,
+                    input_data_format=input_data_format,
+                )
+                # video 에 대해서는 global image (thumbnail) 를 사용하지 않음
+                if not is_video:
+                    image_grids = [anyres_global_image] + image_grids
+            else:
+                image_grids = [image]
+
+            pixel_values = self._preprocess(
+                image_grids,
+                do_resize=do_resize,
+                size=size,
+                resample=resample,
+                do_center_crop=do_center_crop,
+                crop_size=crop_size,
+                do_rescale=do_rescale,
+                rescale_factor=rescale_factor,
+                do_normalize=do_normalize,
+                image_mean=image_mean,
+                image_std=image_std,
+                data_format=data_format,
+                input_data_format=input_data_format,
+            )
+
+            pixel_values = np.array(pixel_values)
+            new_images.append(pixel_values)
+
+            vision_query_length = determine_anyres_num_vision_patches(
+                image_size=image_size,
+                grid_size=crop_size["height"],
+                patch_size=patch_size,
+                possible_resolutions=possible_resolutions,
+                anyres=anyres,
+                unpad=unpad,
+                num_queries_vis_abstractor=num_queries_vis_abstractor,
+                num_queries_vis_abstractor_slow=num_queries_vis_abstractor_slow,
+                is_video=is_video,
+                first_last_frames_slow=first_last_frames_slow,
+                is_first_or_last_frames=is_first_or_last_frames,
+            )
+
+            vision_query_lengths.append(vision_query_length)
+
+        if return_dummy_image:
+            vision_query_lengths = []
+
+        data = {
+            "pixel_values": [torch.tensor(new_image) for new_image in new_images],
+            "image_sizes": [{"width": image_size[1], "height": image_size[0]} for image_size in image_sizes],
+            "vision_query_lengths": vision_query_lengths,
+        }
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def save_pretrained(
+        self,
+        save_directory: Union[str, os.PathLike],
+        *args,
+        **kwargs,
+    ):
+        self.register_for_auto_class()
+        super().save_pretrained(save_directory, *args, **kwargs)
+
+
+def determine_anyres_num_vision_patches(
+    image_size,
+    grid_size,
+    patch_size,
+    possible_resolutions,
+    anyres=False,
+    unpad=True,
+    num_queries_vis_abstractor=0,
+    num_queries_vis_abstractor_slow=0,
+    is_video=False,
+    first_last_frames_slow=False,  # sample-wise option
+    is_first_or_last_frames=False,  # grid-wise option
+):
+    """
+    Computes the number of visual tokens (patches) based on image resolution, grid configuration, and patch size.
+
+    This function supports both fixed-size and any-resolution settings, as well as video-specific configurations
+    such as handling slow frames and frame position flags.
+
+    Args:
+        num_grids (int): Number of grids per image (e.g., 1 for 1x1, 4 for 2x2, etc.).
+        image_size (tuple): The original image size as (height, width).
+        grid_size (int): Size of each grid in pixels (e.g., 336).
+        patch_size (int): Size of each vision patch (e.g., 14 for ViT models).
+        possible_resolutions (list): List of possible resolution tuples [(h1, w1), (h2, w2), ...].
+        anyres (bool, optional): Whether to use any-resolution mode. Defaults to False.
+        unpad (bool, optional): Whether to unpad the image before computing patches. Defaults to True.
+        num_queries_vis_abstractor (int, optional): Number of query tokens for vision abstractor (fast path).
+        num_queries_vis_abstractor_slow (int, optional): Number of query tokens for vision abstractor (slow path).
+        is_video (bool, optional): Whether the input is a video. Defaults to False.
+        first_last_frames_slow (bool, optional): Whether to treat first/last video frames as "slow". Defaults to False.
+        is_first_or_last_frames (bool, optional): Whether current grid corresponds to first/last frame. Defaults to False.
+
+    Returns:
+        int: Total number of visual tokens (patches) after processing.
+    """
+
+    if not anyres:
+        return num_queries_vis_abstractor if num_queries_vis_abstractor > 0 else (grid_size // patch_size) ** 2
+
+    if num_queries_vis_abstractor > 0:
+        num_patch_per_grid = int(num_queries_vis_abstractor**0.5)
+    else:
+        num_patch_per_grid = grid_size // patch_size
+
+    num_global_per_grid = num_patch_per_grid
+
+    # In anyres mode, a global image is included, so there are always at least 2 grids.
+    # However, for video inputs, there is no global image, so it's possible to have only 1 grid.
+    # Therefore, the assertion below is commented out:
+    # assert num_grids > 1
+
+    # Compute the number of vision patches.
+    height, width = select_best_resolution(image_size, possible_resolutions)
+
+    num_patch_height = (height // grid_size) * num_patch_per_grid
+    num_patch_width = (width // grid_size) * num_patch_per_grid
+
+    # local images
+    if unpad:
+        original_height, original_width = image_size
+
+        original_aspect_ratio = original_width / original_height
+        current_aspect_ratio = num_patch_width / num_patch_height
+
+        if original_aspect_ratio > current_aspect_ratio:
+            scale_factor = num_patch_width / original_width
+            new_height = int(original_height * scale_factor)
+            padding = (num_patch_height - new_height) // 2
+            num_patch_height = num_patch_height - padding * 2
+        else:
+            scale_factor = num_patch_height / original_height
+            new_width = int(original_width * scale_factor)
+            padding = (num_patch_width - new_width) // 2
+            num_patch_width = num_patch_width - padding * 2
+
+        num_patches = num_patch_width * num_patch_height + num_patch_height
+    else:
+        num_patches = num_patch_width * num_patch_height
+
+    # In the "slow" strategy, when applying to first and last frames only, it is applied exclusively to those two frames.
+    if num_queries_vis_abstractor_slow > 0:
+        if first_last_frames_slow:
+            if is_first_or_last_frames:
+                num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
+        else:
+            num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
+        # The slowfast feature is only applicable when unpad is set to False.
+        assert unpad is False
+
+    # Global image is not included for video inputs.
+    if not is_video:
+        num_patches += num_global_per_grid**2
+
+    return num_patches
+
+
+def divide_to_grids(image: np.array, grid_size: int, input_data_format=None) -> List[np.array]:
+    """
+    Divides a local image into grids of size (grid_size x grid_size).
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        grid_size (int): The size (in pixels) of each square grid.
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        List[np.array]: A list of image patches, each of size (grid_size x grid_size).
+    """
+    grids = []
+    height, width = get_image_size(image, channel_dim=input_data_format)
+    for i in range(0, height, grid_size):
+        for j in range(0, width, grid_size):
+            if input_data_format == ChannelDimension.LAST:
+                grid = image[i : i + grid_size, j : j + grid_size]
+            else:
+                grid = image[:, i : i + grid_size, j : j + grid_size]
+            grids.append(grid)
+
+    return grids
+
+
+def pad(
+    image: np.array,
+    target_size: tuple,
+    background_color=(127, 127, 127),
+    input_data_format=None,
+) -> np.array:
+    """
+    Pads the input image on the sides (top/bottom and left/right) to match the target height and width.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        target_size (tuple): Target size as (target_height, target_width).
+        background_color (tuple, optional): RGB color value used for padding. Defaults to (127, 127, 127).
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        np.array: The padded image with the specified target size.
+    """
+    target_height, target_width = target_size
+    height, width = get_image_size(image, channel_dim=input_data_format)
+
+    # result = np.ones((target_height, target_width, image.shape[2]), dtype=image.dtype) * background_color
+    result = np.empty((target_height, target_width, image.shape[2]), dtype=image.dtype)
+    for i in range(image.shape[2]):
+        result[..., i].fill(background_color[i])
+
+    paste_x = (target_width - width) // 2
+    paste_y = (target_height - height) // 2
+
+    result[paste_y : paste_y + height, paste_x : paste_x + width, :] = image
+
+    return result
+
+
+def expand2square(
+    image: np.array,
+    bboxes_dict=None,
+    background_color=(127, 127, 127),
+    input_data_format=None,
+) -> np.array:
+    """
+    Expands the input image to a square shape by placing it at the center of a new square canvas,
+    with padding added to the shorter side (either top/bottom or left/right).
+
+    The image is always centered on the new canvas, and padding is applied symmetrically.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        bboxes_dict (dict, optional): A dictionary of bounding boxes, where each value is an NDArray of shape (N, 4, 2)
+            with box coordinates in the format [[xtl, ytl], [xtr, ytr], [xbr, ybr], [xbl, ybl]].
+            Supports multiple categories (e.g., "ocr", "html") simultaneously.
+        background_color (tuple, optional): RGB color to fill the padding area. Defaults to (127, 127, 127).
+        input_data_format (optional): Optional format specifier for image data (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        np.array: A square-shaped image with the original image centered and padded as needed.
+
+    Example:
+        >>> _img = np.ones((80, 100), dtype=np.uint8) * 100
+        >>> _bboxes_dict = {"words": np.array([[[10, 10], [20, 10], [20, 20], [10, 20]],
+        ...                                    [[30, 30], [40, 30], [40, 40], [30, 40]]])}
+        >>> _img, _bboxes_dict = expand2square(_img, _bboxes_dict, (255, 255, 255))
+        >>> _img.shape
+        (100, 100)
+        >>> guessed_ocr_bboxes = np.array([[[20, 10], [30, 10], [30, 20], [20, 20]],
+        ...                                [[40, 30], [50, 30], [50, 40], [40, 40]]])
+        >>> np.testing.assert_array_almost_equal(_bboxes_dict["words"], guessed_ocr_bboxes) is None
+        True
+    """
+    height, width = get_image_size(image, channel_dim=input_data_format)
+    if width == height:
+        return image, bboxes_dict
+    elif width > height:
+        # result = np.ones((width, width, image.shape[2]), dtype=image.dtype) * background_color
+        result = np.empty((width, width, image.shape[2]), dtype=image.dtype)
+        for i in range(image.shape[2]):
+            result[..., i].fill(background_color[i])
+
+        result[(width - height) // 2 : (width - height) // 2 + height, :] = image
+        if bboxes_dict is not None:
+            for key in bboxes_dict:
+                bboxes_dict[key][:, :, 1] += (width - height) // 2
+        return result, bboxes_dict
+    else:
+        # result = np.ones((height, height, image.shape[2]), dtype=image.dtype) * background_color
+        result = np.empty((height, height, image.shape[2]), dtype=image.dtype)
+        for i in range(image.shape[2]):
+            result[..., i].fill(background_color[i])
+
+        result[:, (height - width) // 2 : (height - width) // 2 + width] = image
+        if bboxes_dict is not None:
+            for key in bboxes_dict:
+                bboxes_dict[key][:, :, 0] += (height - width) // 2
+        return result, bboxes_dict
+
+
+def resize_longside(
+    image: np.array,
+    size: int,
+    resample: PILImageResampling = PILImageResampling.BICUBIC,  # type: ignore
+    data_format: Optional[Union[str, ChannelDimension]] = None,
+    input_data_format: Optional[Union[str, ChannelDimension]] = None,
+):
+    """
+    Resizes the image so that its longer side matches the specified size, maintaining the original aspect ratio.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        size (int): Target size for the longer side of the image.
+        resample (PILImageResampling, optional): Resampling method to use during resizing. Defaults to BICUBIC.
+        data_format (str or ChannelDimension, optional): Output data format (e.g., "channels_first" or "channels_last").
+        input_data_format (str or ChannelDimension, optional): Input data format of the image.
+
+    Returns:
+        np.array: The resized image with its aspect ratio preserved.
+    """
+    height, width = get_image_size(image, channel_dim=input_data_format)
+
+    if width == height:
+        target_height, target_width = size, size
+    elif width > height:
+        target_width = size
+        target_height = math.ceil(height / width * size)
+    else:
+        target_width = math.ceil(width / height * size)
+        target_height = size
+
+    return resize(
+        image,
+        size=(target_height, target_width),
+        resample=resample,
+        data_format=data_format,
+        input_data_format=input_data_format,
+    )
+
+
+def _get_local_grids_output_size(image: np.array, target_resolution: tuple, input_data_format=None):
+    """
+    Computes the number of local grids (patches) along the height and width when resizing an image
+    to the target resolution.
+
+    Args:
+        image (np.array): Input image as a NumPy array.
+        target_resolution (tuple): Target resolution in the format (target_height, target_width).
+        input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
+
+    Returns:
+        tuple: A tuple (grid_h, grid_w) representing the number of grids along the height and width.
+    """
+    original_height, original_width = get_image_size(image, channel_dim=input_data_format)
+    target_height, target_width = target_resolution
+
+    scale_w = target_width / original_width
+    scale_h = target_height / original_height
+
+    if scale_w < scale_h:
+        new_width = target_width
+        new_height = min(math.ceil(original_height * scale_w), target_height)
+    else:
+        new_height = target_height
+        new_width = min(math.ceil(original_width * scale_h), target_width)
+
+    return new_height, new_width
+
+
+def select_best_resolution(original_size: tuple, possible_resolutions: list) -> tuple:
+    """
+    Selects the best-fit resolution from a list of possible resolutions based on the original image size.
+
+    This function, adapted from LLaVA-Next
+    (https://github.com/huggingface/transformers/blob/v4.40.2/src/transformers/models/llava_next/image_processing_llava_next.py),
+    evaluates each resolution by computing its effective and wasted area compared to the original size.
+    The optimal resolution is the one that maximizes the effective area while minimizing unused (wasted) space.
+
+    Args:
+        original_size (tuple): The original image size in the format (height, width).
+        possible_resolutions (list): A list of candidate resolutions in the format [(height1, width1), (height2, width2), ...].
+
+    Returns:
+        tuple: The best-fit resolution in the format (height, width).
+    """
+    original_height, original_width = original_size
+    best_fit = None
+    max_effective_resolution = 0
+    min_wasted_resolution = float("inf")
+
+    for height, width in possible_resolutions:
+        scale = min(width / original_width, height / original_height)
+        downscaled_width, downscaled_height = int(original_width * scale), int(original_height * scale)
+        effective_resolution = min(downscaled_width * downscaled_height, original_width * original_height)
+        wasted_resolution = (width * height) - effective_resolution
+
+        if effective_resolution > max_effective_resolution or (
+            effective_resolution == max_effective_resolution and wasted_resolution < min_wasted_resolution
+        ):
+            max_effective_resolution = effective_resolution
+            min_wasted_resolution = wasted_resolution
+            best_fit = (height, width)
+
+    return best_fit

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:340b93b87f93c98b62d2c96ef56e4656d9d68ec8a1cd178fe6812c925f8d8d88
+size 4997245472

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f85a5d24cadbb3d235c670a88f9b0757ff50b226819ba0a3cece51a72a2891e4
+size 4920253536

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4efe99ddd60fc020ad202df4ed089ab0a280c7b17d5376f011888fde8dd2c44
+size 4967583384

model.safetensors.index.json

@@ -0,0 +1,829 @@
+{
+  "metadata": {
+    "total_size": 14884974080
+  },
+  "weight_map": {
+    "image_newline": "model-00001-of-00003.safetensors",
+    "language_model.model.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.19.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "language_model.model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.5.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.norm.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b1.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b2.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.0.b3.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b1.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b2.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv1.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv1.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv1.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv2.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv2.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv2.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv3.bn.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv3.bn.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.conv3.conv.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.se.fc1.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.se.fc1.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.se.fc2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.net.2.b3.se.fc2.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.pos_emb": "model-00003-of-00003.safetensors",
+    "mm_projector.readout.0.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.readout.0.weight": "model-00003-of-00003.safetensors",
+    "mm_projector.readout.2.bias": "model-00003-of-00003.safetensors",
+    "mm_projector.readout.2.weight": "model-00003-of-00003.safetensors",
+    "vision_model.vision_model.embeddings.patch_embedding.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.embeddings.patch_embedding.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.embeddings.position_embedding.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.11.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.12.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.13.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.14.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.15.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.16.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.17.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.18.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.19.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.20.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.21.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.22.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.23.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.24.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.25.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.26.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.layer_norm1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.layer_norm1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.layer_norm2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.layer_norm2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.encoder.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.attention.in_proj_bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.attention.in_proj_weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.attention.out_proj.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.attention.out_proj.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.layernorm.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.layernorm.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.mlp.fc1.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.mlp.fc1.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.mlp.fc2.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.mlp.fc2.weight": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.head.probe": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.post_layernorm.bias": "model-00001-of-00003.safetensors",
+    "vision_model.vision_model.post_layernorm.weight": "model-00001-of-00003.safetensors"
+  }
+}

modeling_hyperclovax.py

@@ -0,0 +1,1344 @@
+import ast
+import contextlib
+import gc
+import json
+import os
+from dataclasses import dataclass
+from functools import partial
+from itertools import chain
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+from einops import rearrange
+from timm.layers import LayerNorm, LayerNorm2d
+from timm.models.regnet import RegStage
+from torch.nn import CrossEntropyLoss
+from transformers import (
+    AutoConfig,
+    AutoModel,
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    PreTrainedModel,
+)
+from transformers.generation.utils import GenerationMixin
+from transformers.modeling_utils import (
+    is_fsdp_enabled,
+    is_local_dist_rank_0,
+    no_init_weights,
+)
+from transformers.models.auto import CONFIG_MAPPING
+from transformers.utils import ModelOutput
+
+from .configuration_hyperclovax import HCXVisionConfig
+from .image_processing_hyperclovax import select_best_resolution
+
+EOT = "<|endofturn|>"
+IMAGE_LOC = "<|dummy3|>"
+VIDEO_LOC = "<|_unuse_missing_100270|>"
+
+
+def get_rank():
+    if dist.is_initialized():
+        return dist.get_rank()
+    return 0
+
+
+def get_world_size():
+    if torch.distributed.is_initialized():
+        world_size = torch.distributed.get_world_size()
+    else:
+        world_size = 1
+    return world_size
+
+
+def unpad_image(tensor: torch.Tensor, original_size: Tuple[int, int]) -> torch.Tensor:
+    """Unpads a PyTorch tensor of a padded and resized image.
+
+    This function removes padding from a tensor image that was previously padded and resized.
+    The padding is removed based on the aspect ratio difference between the original and current image dimensions.
+
+    Args:
+        tensor: The image tensor, assumed to be in CxHxW format.
+        original_size: The original size of the image as (width, height).
+
+    Returns:
+        The unpadded image tensor.
+
+    Examples:
+        >>> import torch
+        >>> # Example 1: Unpadding with height padding
+        >>> padded_tensor = torch.randn(1, 64, 48)  # Padded tensor (C=1, H=64, W=48)
+        >>> original_size = (32, 32)  # Original size (width=32, height=32)
+        >>> unpadded_tensor = unpad_image(padded_tensor, original_size)
+        >>> unpadded_tensor.shape
+        torch.Size([1, 48, 48])
+        >>> # Example 2: Unpadding with width padding
+        >>> padded_tensor = torch.randn(1, 48, 64)  # Padded tensor (C=1, H=48, W=64)
+        >>> original_size = (32, 32)  # Original size (width=32, height=32)
+        >>> unpadded_tensor = unpad_image(padded_tensor, original_size)
+        >>> unpadded_tensor.shape
+        torch.Size([1, 48, 48])
+    """
+    original_width, original_height = original_size
+    current_height, current_width = tensor.shape[1:]
+
+    original_aspect_ratio = original_width / original_height
+    current_aspect_ratio = current_width / current_height
+
+    if original_aspect_ratio > current_aspect_ratio:
+        scale_factor = current_width / original_width
+        new_height = int(original_height * scale_factor)
+        padding = (current_height - new_height) // 2
+        unpadded_tensor = tensor[:, padding : current_height - padding, :]
+    else:
+        scale_factor = current_height / original_height
+        new_width = int(original_width * scale_factor)
+        padding = (current_width - new_width) // 2
+        unpadded_tensor = tensor[:, :, padding : current_width - padding]
+
+    return unpadded_tensor
+
+
+def get_anyres_image_grid_shape(
+    image_size: Tuple[int, int],
+    grid_pinpoints: Union[str, List[Tuple[int, int]]],
+    patch_size: int,
+) -> Tuple[int, int]:
+    """Calculates the image patch grid shape after any-resolution preprocessing.
+
+    Selects the optimal resolution from predefined grid pinpoints based on input image
+    dimensions using `select_best_resolution`, then computes the grid layout by
+    dividing the selected resolution by the patch size using integer division.
+
+    Args:
+        image_size (Tuple[int, int]): Original image dimensions in (width, height) format.
+        grid_pinpoints (Union[str, List[Tuple[int, int]]]): Accepts either:
+            - List of (height, width) resolution tuples
+            - String representation of list (e.g., "[(224, 224), (336, 336)]")
+        patch_size (int): Spatial dimension of square patches for grid division.
+
+    Returns:
+        Tuple[int, int]: Grid dimensions as (num_patches_width, num_patches_height).
+
+    Examples:
+        >>> # Basic case with list input
+        >>> get_anyres_image_grid_shape((1000, 800), [(224, 224), (448, 448)], 112)
+        (4, 4)
+
+        >>> # Basic case with string input
+        >>> get_anyres_image_grid_shape((600, 400), "[(336, 336), (672, 672)]", 112)
+        (6, 6)
+
+        >>> # Case where resolution is not perfectly divisible by patch_size
+        >>> # select_best_resolution picks (224, 224). 224 // 100 = 2
+        >>> get_anyres_image_grid_shape((500, 500), [(224, 224)], 100)
+        (2, 2)
+
+        >>> # Different patch size
+        >>> # select_best_resolution picks (448, 448). 448 // 224 = 2
+        >>> get_anyres_image_grid_shape((1200, 900), [(448, 448), (224, 224)], 224)
+        (2, 2)
+
+    Note:
+        String-formatted grid_pinpoints are converted via ast.literal_eval. Invalid formats
+        may raise syntax exceptions. The actual resolution selection depends on the
+        implementation of `select_best_resolution`. The doctests assume
+        `select_best_resolution` picks the *first* resolution provided in `grid_pinpoints`.
+    """
+    possible_resolutions = grid_pinpoints if isinstance(grid_pinpoints, list) else ast.literal_eval(grid_pinpoints)
+
+    original_width, original_height = image_size
+    height, width = select_best_resolution((original_height, original_width), possible_resolutions)
+    return width // patch_size, height // patch_size
+
+
+def reshape_and_unpad_image_features(
+    image_feature: torch.Tensor,
+    height: int,
+    width: int,
+    image_size: Tuple[int, int],
+    possible_resolutions: List[Tuple[int, int]],
+    grid_size: int,
+    unpad: bool,
+    image_newline: torch.Tensor,
+) -> torch.Tensor:
+    """Reshapes and processes image features with optional unpadding operation.
+
+    Processes input image features by:
+    1. Separating base features from spatial features
+    2. Reshaping spatial features into a 5D tensor (num_patch_height, num_patch_width, height, width, channels)
+    3. Performing either unpadding operation or simple reshaping based on 'unpad' flag
+    4. Concatenating processed features with base features
+
+    Args:
+        image_feature: Input tensor containing image features with shape
+            [1 + num_patches, feature_dim] where the first element is the base feature
+        height: Original image height in pixels
+        width: Original image width in pixels
+        image_size: Target image size as (width, height) tuple
+        possible_resolutions: List of possible [height, width] resolutions for multi-scale processing
+        grid_size: Grid dimension for patch arrangement
+        unpad: Flag to enable unpadding operation
+        image_newline: Special token tensor used as separator when unpadding
+
+    Returns:
+        torch.Tensor: Processed image features tensor with shape [1 + num_processed_patches, feature_dim]
+
+    Raises:
+        AssertionError: If base feature dimension doesn't match height*width
+    """
+    base_image_feature = image_feature[0]
+    image_feature = image_feature[1:]
+
+    assert (
+        height * width == base_image_feature.shape[0]
+    ), f"height: {height}, width: {width}, base_image_feature.shape[0]: {base_image_feature.shape[0]}"
+
+    num_patch_width, num_patch_height = get_anyres_image_grid_shape(image_size, possible_resolutions, grid_size)
+    image_feature = image_feature.view(num_patch_height, num_patch_width, height, width, -1)
+
+    if unpad:
+        image_feature = image_feature.permute(4, 0, 2, 1, 3).contiguous()
+        image_feature = image_feature.flatten(1, 2).flatten(2, 3)
+        image_feature = unpad_image(image_feature, image_size)
+        image_feature = torch.cat(
+            (
+                image_feature,
+                image_newline[:, None, None].expand(*image_feature.shape[:-1], 1).to(image_feature.device),
+            ),
+            dim=-1,
+        )
+        image_feature = image_feature.flatten(1, 2).transpose(0, 1)
+    else:
+        image_feature = image_feature.permute(0, 2, 1, 3, 4).contiguous()
+        image_feature = image_feature.flatten(0, 3)
+    image_feature = torch.cat((base_image_feature, image_feature), dim=0)
+
+    return image_feature
+
+
+def anyres_postprocessing(
+    image_forward_outs: List[torch.FloatTensor],
+    image_sizes: List[List[int]],
+    possible_resolutions: List[Tuple[int, int]],
+    patch_size: int,
+    grid_size: int,
+    image_newline: torch.FloatTensor,
+    num_queries_vis_abstractor: int = -1,
+    unpad: bool = False,
+) -> List[torch.FloatTensor]:
+    """Processes 2D visual features into 1D sequences with post-processing steps.
+
+    Performs AnyRes postprocessing by flattening 2D visual features from grid partitions into 1D sequences, adding
+    newline embeddings at row boundaries for images, and optionally removing padding regions based on original image
+    sizes. For video data, processes each frame's features separately into a single sequence per video and disables
+    unpadding and newline insertion.
+
+    Args:
+        image_forward_outs (List[torch.FloatTensor]): List of input tensors with shape
+            (number_of_images_in_grid, total_patches, feature_dim) containing visual features.
+        split_sizes (List[int]): A list containing the number of patches for each sample in the batch. The sum of
+            `split_sizes` should equal `image_forward_outs.shape[0]`.
+        image_sizes (List[List[int]]): A list where each element is a list `[width, height]` representing the original
+            dimensions of the corresponding image sample. Used for unpadding.
+        possible_resolutions (List[Tuple[int, int]]): A list of supported resolution tuples `(height, width)` used by
+            `reshape_and_unpad_image_features` for spatial reconstruction, especially during unpadding.
+        patch_size (int): The spatial dimension (height and width) of the square patches the image was divided into.
+        grid_size (int): The spatial dimension (height and width) of the square grid onto which patches are mapped.
+            `grid_size` should be divisible by `patch_size`.
+        image_newline (torch.FloatTensor): A learnable tensor representing the newline embedding, typically with shape
+            (1, feature_dim). Added after each row of image patches when not unpadding.
+        num_queries_vis_abstractor (int, optional): If a visual abstractor with a fixed number of output queries is used
+            instead of grid patching, this specifies the number of queries. Must be a perfect square if > 0.
+            Defaults to -1 (indicating standard grid patching is used).
+        unpad (bool, optional): If `True`, removes padding tokens from image features based on `image_sizes` and
+            `possible_resolutions`. Does not apply to video features. Defaults to False.
+
+    Returns:
+        List[torch.FloatTensor]: A list of tensors, where each tensor represents the processed 1D sequence of visual
+            features for a single sample from the input batch. The length of the sequence varies depending on processing
+            (unpadding, newlines, video flattening).
+
+    Raises:
+        AssertionError: If `num_queries_vis_abstractor` is greater than 0 but not a perfect square.
+    """
+    height = width = grid_size // patch_size
+
+    if num_queries_vis_abstractor > 0:
+        assert (num_queries_vis_abstractor**0.5).is_integer(), "n_queries must be square number"
+        height = width = int(num_queries_vis_abstractor**0.5)
+
+    # post-processing (unpad, add newline)
+    new_image_features = []
+    for image_idx, image_feature in enumerate(image_forward_outs):
+        if image_feature.shape[0] > 1:
+            image_feature = reshape_and_unpad_image_features(
+                image_feature=image_feature,
+                height=height,
+                width=width,
+                image_size=image_sizes[image_idx],
+                possible_resolutions=possible_resolutions,
+                grid_size=grid_size,  # Pass grid info if needed by helper
+                unpad=unpad,
+                image_newline=image_newline,
+            )
+        else:
+            image_feature = image_feature[0]
+            image_feature = torch.cat((image_feature, image_newline[None].to(image_feature.device)), dim=0)
+        new_image_features.append(image_feature)
+    image_features = new_image_features
+    return image_features
+
+
+@dataclass
+class HCXVisionOutput(ModelOutput):
+    """Output class for vision models, containing various computation results.
+
+    Args:
+        loss (Optional[torch.FloatTensor], optional): Total cross-entropy loss calculated from logits and labels.
+        loss_per_sample (Optional[torch.FloatTensor], optional): Per-sample loss values for advanced loss processing.
+        logits (torch.FloatTensor): Classification scores (before SoftMax) of shape (batch_size, num_classes).
+        past_key_values (Optional[Tuple[Tuple[torch.FloatTensor]]], optional): Contains precomputed hidden-states
+            that can be used (see `past_key_values` input) to speed up sequential decoding.
+        hidden_states (Optional[Tuple[torch.FloatTensor]], optional):
+            Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
+            shape (batch_size, sequence_length, hidden_size).
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (Optional[Tuple[torch.FloatTensor]], optional): Tuple of torch.FloatTensor (one for each layer)
+            of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention
+            softmax, used to compute the weighted average in the self-attention heads.
+    """
+
+    loss: Optional[torch.FloatTensor] = None
+    loss_per_sample: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
+class HCXVisionForCausalLM(PreTrainedModel, GenerationMixin):
+    """HCX Vision model for causal language modeling with vision-language capabilities.
+
+    This class combines a vision model with a language model to create a multimodal model
+    capable of processing images or videos and generating text based on the visual inputs.
+
+    Attributes:
+        config_class: Configuration class for the model.
+        vision_model_name: Name of the vision model component.
+        _no_split_modules: List of modules that should not be split during parallel processing.
+        supports_gradient_checkpointing: Whether the model supports gradient checkpointing.
+        _skip_keys_device_placement: Keys to skip during device placement.
+    """
+
+    config_class = HCXVisionConfig
+    vision_model_name = "vision_model"
+    _no_split_modules = ["SiglipEncoderLayer", "LlamaDecoderLayer", "HyperCLOVAXDecoderLayer"]
+    supports_gradient_checkpointing = True
+    _skip_keys_device_placement = "past_key_values"
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+
+    def __init__(
+        self,
+        config: HCXVisionConfig,
+        **kwargs: Optional[Any],
+    ) -> None:
+        """Initialize the HCXVisionForCausalLM model.
+
+        Args:
+            config: Configuration object for the model containing parameters for both
+                vision and language components.
+            **kwargs: Additional keyword arguments:
+                - use_liger: Whether to use liger kernel for hyperclovax models.
+                - use_fused_ce: Whether to use fused cross-entropy loss.
+                - use_sum_loss: Whether to use sum reduction for loss instead of mean.
+                - is_safetensor_save: Whether to save model using safetensors format.
+
+        Raises:
+            ValueError: If vision_config is not defined or if text_config is not defined.
+        """
+        super().__init__(config)  # self.config = config
+
+        # init configs
+        text_config = self._init_text_config(config)
+        vision_config = self._init_vision_config(config)
+        
+        ## possible_resolution should be matched with preprocessor_config.json
+        config.possible_resolutions = self._init_possible_resolutions(config, vision_config)
+
+        # init models & parameters
+        with no_init_weights():  # weight will be loaded in from_pretrained
+            self.vision_model = AutoModel.from_config(vision_config, trust_remote_code=True)
+
+        self.mm_projector = self._init_mm_projector(config, text_config, vision_config)
+
+        self.language_model = AutoModelForCausalLM.from_config(text_config)
+        self.lm_head_vocab_size = getattr(text_config, "padded_vocab_size", text_config.vocab_size)
+        self.language_model.lm_head = nn.Linear(text_config.hidden_size, self.lm_head_vocab_size, bias=False)
+
+        if config.anyres:
+            self.image_newline = nn.Parameter(torch.empty(text_config.hidden_size, dtype=self.dtype))
+
+        # modify configs or model settings
+        if text_config.model_type in ["llama", "hyperclovax", "gpt2"]:
+            self.language_model.gradient_checkpointing_enable()
+        if text_config.model_type == "hyperclovax" and self.use_liger:
+            self.language_model._get_apply_liger_kernel_converter()(model=self.language_model)
+
+        # update configs
+        self.vision_config = vision_config = self.vision_model.config
+        self.text_config = text_config = self.language_model.config
+        config.update({"vision_config": vision_config})
+        config.update({"text_config": text_config})
+
+        # etc
+        self.use_liger = kwargs.pop("use_liger", False)
+        self.use_fused_ce = kwargs.pop("use_fused_ce", False)
+        self.use_meansum_loss = kwargs.pop("use_meansum_loss", False)
+        self.freeze_before_sampler = kwargs.pop("freeze_before_sampler", False)
+        self.use_turnmeansum_loss = kwargs.pop("use_turnmeansum_loss", False)
+        self.vision_input_chunk_size = kwargs.pop("vision_input_chunk_size", None)
+        self.is_safetensor_save = kwargs.get("is_safetensor_save", True)
+
+        use_sum_loss = True if kwargs.pop("use_sum_loss", False) else False
+        self.reduction = self._init_reduction_type(use_sum_loss)
+
+        self.vision_model_use_no_grad = None  # forward 시 체크 및 할당
+
+        self._backward_compatibility_gradient_checkpointing()  # self.post_init() 에 포함되어 있는 gc 가능한지 확인하고 켜주는 함수
+
+    def _init_weights(self, module):
+        # copies from https://github.com/kakaobrain/honeybee/blob/main/honeybee/common_layers.py#L55
+        if (
+            isinstance(module, nn.Conv2d)  # noqa: SIM101
+            or isinstance(module, nn.Embedding)
+            or isinstance(module, nn.Linear)
+        ):
+            module.weight.data.normal_(mean=0.0, std=0.02)
+            if hasattr(module, "bias") and module.bias is not None:
+                module.bias.data.zero_()
+
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+        elif isinstance(module, nn.Parameter):
+            embed_std = 1 / torch.sqrt(torch.tensor(module.size(0), dtype=torch.float)).to(module.dtype)
+            module.data.normal_(mean=0.0, std=embed_std)
+
+    def _init_reduction_type(self, use_sum_loss):
+        assert not (
+            self.use_meansum_loss and self.use_turnmeansum_loss
+        ), "use_meansum_loss and use_turnmeansum_loss cannot both be True; only one or neither may be True."
+        if self.use_meansum_loss or self.use_turnmeansum_loss:
+            reduction = "none"
+        elif use_sum_loss:
+            reduction = "sum"
+        else:
+            reduction = "mean"
+        return reduction
+
+    def _init_vision_config(self, config):
+        vision_model_type = config.vision_config.model_type
+        if vision_model_type in CONFIG_MAPPING:
+            vision_config = CONFIG_MAPPING[vision_model_type](**config.vision_config.to_dict())
+            vision_config.auto_map = {}
+        else:
+            if config.vision_model_name_or_path is not None:
+                vision_config = AutoConfig.from_pretrained(config.vision_model_name_or_path, trust_remote_code=True)
+            elif config.vision_config._name_or_path is not None:
+                vision_config = AutoConfig.from_pretrained(config.vision_config._name_or_path, trust_remote_code=True)
+            else:
+                raise ValueError("vision_config is not defined")
+
+        vision_config.anyres = config.anyres
+        vision_config.max_num_grids = config.max_num_grids
+        return vision_config
+
+    def _init_text_config(self, config):
+        if hasattr(config, "text_config") and config.text_config is not None:
+            model_type = config.text_config.model_type
+            text_config = CONFIG_MAPPING[model_type](**config.text_config.to_dict())
+        else:
+            raise ValueError("text_config is not defined")
+        text_config._attn_implementation = config._attn_implementation
+        if text_config.model_type != "hyperclovax":
+            text_config.logits_scaling = 1.0
+        return text_config
+
+    def _init_possible_resolutions(self, config, vision_config):
+        """possible_resolution should be matched with preprocessor_config.json"""
+        if not getattr(config, "possible_resolutions", []):
+            possible_resolutions = []
+            if config.anyres:
+                assert config.max_num_grids > 0
+                for i in range(1, config.max_num_grids + 1):
+                    for j in range(1, config.max_num_grids + 1):
+                        if i == 1 and j == 1 and not config.use_1x1_grid:
+                            continue
+                        if i * j <= config.max_num_grids:
+                            possible_resolutions.append([i, j])
+
+                possible_resolutions = [
+                    [ys * vision_config.image_size, xs * vision_config.image_size] for ys, xs in possible_resolutions
+                ]
+            return possible_resolutions
+        else:
+            return config.possible_resolutions
+
+    def _init_mm_projector(self, config, text_config, vision_config):
+        input_hidden_size = vision_config.hidden_size
+        if config.mm_projector_type == "linear":
+            mm_projector = nn.Linear(input_hidden_size, text_config.hidden_size)
+            mm_projector.dtype = next(mm_projector.parameters()).dtype
+        elif config.mm_projector_type == "cabstractor":
+            mm_projector = HCXVisionCAbstractor(
+                num_queries=config.num_queries_vis_abstractor_image,
+                num_input_tokens=(vision_config.image_size // vision_config.patch_size) ** 2,
+                encoder_hidden_size=input_hidden_size,
+                hidden_size=input_hidden_size,
+                output_hidden_size=text_config.hidden_size,
+                pos_emb=config.proj_pos_emb,
+                prenorm=config.proj_prenorm,
+            )
+        else:
+            mm_projector = HCXVisionMlp(
+                config.mm_projector_type,
+                input_hidden_size,
+                hidden_features=input_hidden_size,  # TODO: llava 처럼 hidden_size 를 input_hidden_size 가 아니라 LLM embedding size 로 바꿔주기
+                out_features=self.text_config.hidden_size,
+            )
+        return mm_projector
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values_images: Optional[List[List[torch.FloatTensor]]] = None,
+        image_sizes_images: Optional[List[List[Tuple[int, int]]]] = None,
+        pixel_values_videos: Optional[List[List[torch.FloatTensor]]] = None,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> Union[Tuple, HCXVisionOutput]:
+        """Forward pass of the model.
+
+        This method processes the input tokens and images, combines them into a unified
+        representation, and generates text output based on the inputs.
+
+        Args:
+            input_ids: Input token IDs. In positions where images are inputted, the value is replaced by "<|dummy3|>"
+            pixel_values: List of lists of 4D tensors for images. Each outer list corresponds to a batch and contains
+                inner lists of image tensors.
+            past_key_values: Pre-computed key and value states of the attention layers for faster inference.
+            attention_mask: Mask to avoid performing attention on padding token indices.
+            inputs_embeds: Input embeddings. If provided, input_ids will not be used.
+            labels: Labels for computing the language modeling loss.
+            use_cache: Whether to use past key/values for faster inference.
+            output_attentions: Whether to return attention weights of each layer.
+            output_hidden_states: Whether to return hidden states of each layer.
+            return_dict: Whether to return a ModelOutput instead of a tuple.
+            image_sizes: List of lists representing image dimensions (width, height).
+            vision_query_lengths: List of lists containing lengths when each image is converted into visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            num_queries_vis_abstractors: List of lists containing number of visual tokens for each image grid.\
+                For video frames, this is the number of visual tokens for the fast part.
+            num_queries_vis_abstractors_slow: List of lists containing number of visual tokens for
+                the slow part when applying the slowfast algorithm to video frames.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is
+                applied to the first or last frames of the video.
+            is_video_list: List of booleans indicating which inputs are videos.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            If return_dict=True, returns an HCXVisionOutput object containing:
+                - loss: Language modeling loss if labels are provided, otherwise None.
+                - loss_per_sample: Per-sample loss if labels are provided, otherwise None.
+                - logits: Prediction scores of the language modeling head.
+                - past_key_values: Past key/values for faster inference if use_cache=True.
+                - hidden_states: Hidden states of all layers if output_hidden_states=True.
+                - attentions: Attention weights of all layers if output_attentions=True.
+            If return_dict=False, returns a tuple containing the above items except loss_per_sample.
+        """
+        output_attentions = (
+            output_attentions if output_attentions is not None else self.config.vision_config.output_attentions
+        )
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.vision_config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if inputs_embeds is None and past_key_values is None:
+            if pixel_values_images is not None or pixel_values_videos is not None:
+                inputs_embeds = self.extract_inputs_embeds(
+                    input_ids=input_ids,
+                    pixel_values_images=pixel_values_images,
+                    image_sizes_images=image_sizes_images,
+                    pixel_values_videos=pixel_values_videos,
+                )
+            else:
+                inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        if inputs_embeds is not None:
+            input_ids = None
+
+        ################################
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.language_model.base_model(
+            input_ids=input_ids,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        hidden_states = hidden_states * self.text_config.logits_scaling
+
+        loss = None
+        loss_per_sample = None
+        logits = self.language_model.lm_head(hidden_states)
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss(reduction="none")  # ignore IGNORE_INDEX(-100)
+            shift_logits = shift_logits.view(-1, self.lm_head_vocab_size)
+            shift_labels = shift_labels.view(-1)
+
+            # Enable model/pipeline parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+            if get_rank() == 0:
+                loss_per_sample = loss.view(logits.shape[0], -1).sum(axis=1) / (
+                    shift_labels.view(logits.shape[0], -1) != self.config.ignore_index
+                ).sum(axis=1)
+            loss = loss[shift_labels != self.config.ignore_index].mean()
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return HCXVisionOutput(
+            loss=loss,
+            loss_per_sample=loss_per_sample,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_input_embeddings
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_input_embeddings
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_output_embeddings
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_output_embeddings
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.set_decoder
+    def set_decoder(self, decoder):
+        self.language_model.set_decoder(decoder)
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.get_decoder
+    def get_decoder(self):
+        return self.language_model.get_decoder()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.tie_weights
+    def tie_weights(self):
+        return self.language_model.tie_weights()
+
+    # Copied from transformers.models.llava.modeling_llava.LlavaForConditionalGeneration.resize_token_embeddings
+    def resize_token_embeddings(self, new_num_tokens: Optional[int] = None, pad_to_multiple_of=None) -> nn.Embedding:
+        model_embeds = self.language_model.resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
+        self.config.text_config.vocab_size = model_embeds.num_embeddings
+        self.vocab_size = model_embeds.num_embeddings
+        return model_embeds
+
+    def extract_inputs_embeds(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values_images: Optional[List[List[torch.FloatTensor]]] = None,
+        image_sizes_images: Optional[List[List[Tuple[int, int]]]] = None,
+        pixel_values_videos: Optional[List[List[torch.FloatTensor]]] = None,
+    ):
+        """Extract input embeddings by processing text tokens and visual features.
+
+        This method processes the input tokens and image features, extracts the visual features
+        using the vision model, and combines them with the text token embeddings to create
+        a unified input representation for the language model.
+
+        Args:
+            input_ids: Input token IDs with img_start_id markers for image positions.
+            pixel_values: List of lists of image tensors.
+            past_key_values: Pre-computed key and value states for faster inference.
+            image_sizes: List of lists of image dimensions (width, height).
+            vision_query_lengths: List of lists of lengths when each image is converted to visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is
+                applied to the first or last frames of the video.
+            is_videos: List of booleans indicating which inputs are videos.
+
+        Returns:
+            Combined embeddings of text tokens and visual features.
+        """
+        # for convert back to List of List format
+        len_pixel_values_images = [len(pixel_value) for pixel_value in pixel_values_images] if pixel_values_images else []
+        len_pixel_values_videos = [len(pixel_value) for pixel_value in pixel_values_videos] if pixel_values_videos else []
+
+        if sum(len_pixel_values_images) + sum(len_pixel_values_videos) == 0:
+            return None
+
+        inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        if sum(len_pixel_values_images) > 0:
+            image_features_batch = self.forward_images(
+                pixel_values_images, image_sizes_images, len_pixel_values_images
+            )
+            for i, image_features in enumerate(image_features_batch):
+                if len(image_features) > 0:
+                    image_token_indices = (input_ids[i] == self.config.image_token_id).nonzero().squeeze()
+                    inputs_embeds[i][image_token_indices] = torch.cat(image_features).to(inputs_embeds.dtype)
+
+        if sum(len_pixel_values_videos) > 0:
+            video_features_batch = self.forward_videos(pixel_values_videos, len_pixel_values_videos)
+            for i, video_features in enumerate(video_features_batch):
+                if len(video_features) > 0:
+                    video_token_indices = (input_ids[i] == self.config.video_token_id).nonzero().squeeze()
+                    inputs_embeds[i][video_token_indices] = torch.cat(video_features).to(inputs_embeds.dtype)
+
+        return inputs_embeds
+
+    def forward_images(
+        self,
+        pixel_values_images: List[List[torch.FloatTensor]],
+        image_sizes_images: List[List[Tuple[int, int]]],
+        len_pixel_values_images: List[int],
+    ) -> List[List[torch.Tensor]]:
+        if sum(len_pixel_values_images) == 0:
+            return None
+
+        concat_pixel_values_images = torch.cat(list(chain(*pixel_values_images)), dim=0)
+
+        visual_token_idx = 0 if "siglip" in self.vision_config.model_type else 1
+        context_vision_model = torch.no_grad() if self.vision_model_use_no_grad else contextlib.nullcontext()
+        with context_vision_model:
+            if self.config.use_nth_layer == -1:
+                # Replace post_layernorm of the last layer with Identity
+                self.vision_model.vision_model.post_layernorm = nn.Identity()
+                image_forward_outs = self.vision_model(concat_pixel_values_images)
+                image_forward_outs = image_forward_outs.last_hidden_state[:, visual_token_idx:]
+            else:
+                image_forward_outs = self.vision_model(concat_pixel_values_images, output_hidden_states=True)
+                image_forward_outs = image_forward_outs.hidden_states[self.config.use_nth_layer][:, visual_token_idx:]
+
+        image_forward_outs = image_forward_outs.to(dtype=self.mm_projector.dtype)
+        image_forward_outs = self.mm_projector(image_forward_outs)  # b (h w) d
+
+        # feature 를 분할. e.g. torch.Size([18, 81, 3072]) -> [torch.Size([9, 81, 3072]), torch.Size([9, 81, 3072])]
+        split_sizes = [pixel_value.shape[0] for pixel_value in chain(*pixel_values_images)]
+        image_forward_outs = torch.split(image_forward_outs, split_sizes, dim=0)
+
+        # newline 붙여주기 (anyres postprocessing)
+        image_features = anyres_postprocessing(
+            image_forward_outs=image_forward_outs,
+            image_sizes=[image_size for image_sizes in image_sizes_images for image_size in image_sizes],
+            num_queries_vis_abstractor=self.config.num_queries_vis_abstractor_image,
+            unpad=self.config.unpad,
+            patch_size=self.vision_config.patch_size,
+            grid_size=self.vision_config.image_size,
+            image_newline=self.image_newline,
+            possible_resolutions=self.config.possible_resolutions,
+        )
+
+        # 원래 pixel_values_images 형태로 복원
+        image_features = [
+            image_features[sum(len_pixel_values_images[:i]) : sum(len_pixel_values_images[: i + 1])]
+            for i in range(len(len_pixel_values_images))
+        ]
+
+        return image_features
+
+    def forward_videos(
+        self,
+        pixel_values_videos: List[List[torch.FloatTensor]],
+        len_pixel_values_videos: List[int],
+    ) -> List[torch.Tensor]:
+
+        len_video_grids = sum(len_pixel_values_videos)
+        if len_video_grids == 0:
+            return None
+
+        # Run Vision Model
+        concat_pixel_values_videos = torch.cat(list(chain(*pixel_values_videos)), dim=0)
+
+        visual_token_idx = 0 if "siglip" in self.vision_config.model_type else 1
+        context_vision_model = torch.no_grad() if self.vision_model_use_no_grad else contextlib.nullcontext()
+        with context_vision_model:
+            if self.config.use_nth_layer == -1:
+                # Replace post_layernorm of the last layer with Identity
+                self.vision_model.vision_model.post_layernorm = nn.Identity()
+                video_forward_outs = self.vision_model(concat_pixel_values_videos)
+                video_forward_outs = video_forward_outs.last_hidden_state[:, visual_token_idx:]
+            else:
+                video_forward_outs = self.vision_model(concat_pixel_values_videos, output_hidden_states=True)
+                video_forward_outs = video_forward_outs.hidden_states[self.config.use_nth_layer][:, visual_token_idx:]
+
+        video_forward_outs = video_forward_outs.to(dtype=self.mm_projector.dtype)
+
+        # Run MM-Projector
+        # len(num_grids) == len(num_queries_vis_abstractors) + 1
+        grid_idx = 0
+        num_grids = [grid_idx]  # e.g. [0, 9, 18, 19, 27, 28, 36, 37, 45, 46, 54, 55, 56]
+        num_queries_vis_abstractors = []  # e.g. [81, 81, 81, 9, 81, 9, 81, 9, 81, 9, 81, 9]
+        len_total_frames = video_forward_outs.shape[0]
+
+        if self.config.first_last_frames_slow:
+            # TODO: 동작 확인 안 했음. 해야 함.
+            # slowfast (first_last_frames_slow)
+            assert len_total_frames != 0
+            if len_total_frames <= 2:
+                num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_slow)
+                grid_idx += len_total_frames
+                num_grids.append(grid_idx)
+            else:
+                num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_slow)
+                grid_idx += 1
+                num_grids.append(grid_idx)
+
+                num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_fast)
+                grid_idx += len_total_frames - 2
+                num_grids.append(grid_idx)
+
+                num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_slow)
+                grid_idx += 1
+                num_grids.append(grid_idx)
+        else:
+            # slowfast
+            for pixel_values_frames in pixel_values_videos:
+                for pixel_values_frame in pixel_values_frames:
+                    if len(pixel_values_frame) > 0:
+                        num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_slow)
+                        grid_idx += 1
+                        num_grids.append(grid_idx)
+                        num_queries_vis_abstractors.append(self.config.num_queries_vis_abstractor_video_fast)
+                        grid_idx = grid_idx + len(pixel_values_frame) - 1
+                        num_grids.append(grid_idx)
+
+        video_forward_outs = self.mm_projector(video_forward_outs, num_queries_vis_abstractors, num_grids)
+
+        # video_group 별로 concat 처리.
+        # 예를 들어, 3x3 grid 를 사용했을 경우, 총 9개의 feature 가 모일 때까지, grouped_features 에 리스트를 모아주고, concat 처리.
+        video_features = []  # what we want to return
+        target_features = []
+        target_group_size = 0
+        group_counter = 0
+        video_groups = [
+            len(frame) for frames in pixel_values_videos for frame in frames
+        ]  # for concat video features after projector
+
+        for forward_out in video_forward_outs:
+            target_group_size += len(forward_out)
+            target_features.append(forward_out.flatten(0, 1))
+
+            video_group_size = video_groups[group_counter]
+            if video_group_size == target_group_size:
+                video_features.append(torch.cat(target_features, dim=0))
+                target_features = []
+                group_counter += 1
+                target_group_size = 0
+
+            elif video_group_size < target_group_size:
+                raise RuntimeError(f"video_group_size < target_group_size!! [{video_group_size} < {target_group_size}]")
+
+        assert len(target_features) == 0, f"target_features is not empty!! {target_features}"
+        assert len(video_groups) == len(video_features)
+
+        # 원래 pixel_values_videos 형태로 복원
+        video_features = [
+            video_features[sum(len_pixel_values_videos[:i]) : sum(len_pixel_values_videos[: i + 1])]
+            for i in range(len(len_pixel_values_videos))
+        ]
+
+        return video_features
+
+    @torch.no_grad()
+    def generate(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values_images: Optional[List[List[torch.FloatTensor]]] = None,
+        image_sizes_images: Optional[List[List[Tuple[int, int]]]] = None,
+        pixel_values_videos: Optional[List[List[torch.FloatTensor]]] = None,
+        pad_token_id: Optional[int] = None,
+        eos_token_id: Optional[int] = None,
+        bad_words_ids: Optional[List[List[int]]] = None,
+        max_length: int = 196,
+        min_length: int = 2,
+        do_sample: bool = True,
+        num_beams: int = 1,
+        top_p: float = 0.6,
+        top_k: int = 0,
+        temperature: float = 0.5,
+        repetition_penalty: float = 1.0,
+        length_penalty: int = 1,
+        use_cache: bool = True,
+        verbose: bool = False,
+        **kwargs,
+    ) -> torch.LongTensor:
+        """Generate text based on input tokens and images.
+
+        This method generates text based on the provided input tokens and images using
+        beam search and/or sampling strategies.
+
+        Args:
+            input_ids: Input token IDs with img_start_id markers for image positions.
+            pixel_values: List of lists of image tensors.
+            image_sizes: List of lists of image dimensions (width, height).
+            vision_query_lengths: List of lists of lengths when each image is converted to visual tokens.
+            non_vision_query_lengths: List of lengths of text tokens (excluding visual tokens) for each sample.
+            num_queries_vis_abstractors: List of lists containing number of visual tokens for each image grid.
+            num_queries_vis_abstractors_slow: List of lists containing number of visual tokens for the slow part when
+                applying the slowfast algorithm to video frames.
+            first_last_frames_slows: List of booleans indicating whether the slowfast algorithm is applied to the first
+                or last frames of the video.
+            is_videos: List of booleans indicating which inputs are videos.
+            img_start_ids_list: List of lists containing indices of img_start_id tokens for each sample.
+            pad_token_id: Token ID used for padding.
+            eos_token_id: Token ID used to signal the end of a sequence.
+            bad_words_ids: List of token ID sequences that should not be generated.
+            max_length: Maximum length of the sequence to be generated (input length + max_new_tokens).
+            min_length: Minimum length of the sequence to be generated (input length + min_new_tokens).
+            do_sample: Whether to use sampling for generation (otherwise uses greedy decoding).
+            num_beams: Number of beams for beam search. 1 means no beam search.
+            top_p: Nucleus sampling parameter. Tokens with cumulative probability > top_p are kept.
+            top_k: Number of highest probability tokens to keep for top-k-filtering.
+            temperature: Value used to modulate the next token probabilities.
+            repetition_penalty: Penalty applied to tokens that have already appeared in the sequence.
+            length_penalty: Exponential penalty applied to sequence length.
+            use_cache: Whether to use past key/values for faster inference.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            Generated token IDs.
+        """
+        # inputs_embeds: torch.bfloat16 : [batchsize, variable(visual token, text token, system prompt 모두 포함)]
+        if pad_token_id is None:
+            pad_token_id = self.tokenizer.pad_token_id
+        if eos_token_id is None:
+            eos_token_id = self.tokenizer.encode("<|endofturn|>")[0]
+        if bad_words_ids is None:
+            bad_words_ids = [
+                [
+                    self.config.text_config.bos_token_id,
+                ],
+                [
+                    self.config.text_config.eos_token_id,
+                ],
+            ]
+
+        if (pixel_values_images is None or all(len(pixel_values) == 0 for pixel_values in pixel_values_images)) and (
+            pixel_values_videos is None or all(len(pixel_values) == 0 for pixel_values in pixel_values_videos)
+        ):
+            return self.language_model.generate(
+                input_ids, pad_token_id=pad_token_id, eos_token_id=eos_token_id, bad_words_ids=bad_words_ids, **kwargs
+            )
+
+        inputs_embeds = self.extract_inputs_embeds(
+            input_ids=input_ids,
+            pixel_values_images=pixel_values_images,
+            image_sizes_images=image_sizes_images,
+            pixel_values_videos=pixel_values_videos,
+        )
+
+        inputs_embeds = inputs_embeds.to(device=self.language_model.device, dtype=self.language_model.dtype)
+
+        # pred : torch.int64 : [batchsize, generated token_length]
+        pred = self.language_model.generate(
+            inputs_embeds=inputs_embeds,
+            pad_token_id=pad_token_id,
+            eos_token_id=eos_token_id,
+            bad_words_ids=bad_words_ids,
+            max_new_tokens=max_length,
+            min_length=min_length,
+            num_beams=num_beams,
+            do_sample=(False if temperature == 0.0 else do_sample),  # set do_sample=False if invalid temperature
+            top_k=top_k,
+            top_p=top_p,
+            temperature=temperature,
+            repetition_penalty=repetition_penalty,
+            length_penalty=length_penalty,
+            early_stopping=(False if num_beams <= 1 else True),  # set early_stopping=False when not beam_search
+            use_cache=use_cache,
+        )
+
+        if verbose:
+            llm_query = self.tokenizer.batch_decode(
+                [
+                    [token_id for token_id in input_ids_row if token_id != self.tokenizer.pad_token_id]
+                    for input_ids_row in input_ids.detach().cpu().tolist()
+                ],
+                skip_special_tokens=False,
+            )[0]
+            llm_pred = self.tokenizer.batch_decode(
+                [
+                    [token_id for token_id in pred_row if token_id != self.tokenizer.pad_token_id]
+                    for pred_row in pred.detach().cpu().tolist()
+                ],
+                skip_special_tokens=False,
+            )[0]
+            print(f"# [info] llm_query: {llm_query}")
+            print(f"# [info] llm_pred: {llm_pred}")
+
+        return pred
+
+    def to_vision_model_device(self, input_tensor: Union[torch.Tensor, List]) -> Union[torch.Tensor, List]:
+        """Move input tensors to the vision model's device.
+        This method recursively moves input tensors or lists of tensors to the vision model's device.
+
+        Args:
+            input_tensor: Input tensor or list of tensors to be moved to the vision model's device.
+
+        Returns:
+            The input tensor or list of tensors moved to the vision model's device.
+
+        Raises:
+            TypeError: If the input is neither a tensor nor a list.
+        """
+        if isinstance(input_tensor, list):
+            return [self.to_vision_model_device(item) for item in input_tensor]
+        elif isinstance(input_tensor, torch.Tensor):
+            return input_tensor.to(self.vision_model.device)
+        else:
+            raise TypeError("Unsupported data type. Only tensors and lists are allowed.")
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids: torch.LongTensor,
+        past_key_values: Optional[Tuple[Tuple[torch.Tensor]]] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        **kwargs,
+    ) -> Dict[str, Any]:
+        """Prepare inputs for the generation algorithm.
+
+        This method prepares the input for each generation step based on the model's needs.
+
+        Args:
+            input_ids: Input token IDs.
+            past_key_values: Pre-computed key and value states for faster inference.
+            attention_mask: Mask to avoid performing attention on padding token indices.
+            inputs_embeds: Input embeddings. If provided, input_ids will not be used.
+            **kwargs: Additional keyword arguments.
+
+        Returns:
+            Dictionary containing the prepared inputs for the model.
+        """
+        input_ids = kwargs.get("decoder_input_ids", input_ids)
+
+        if past_key_values:
+            input_ids = input_ids[:, -1:]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "pixel_values": kwargs.get("pixel_values", None),
+            }
+        )
+        return model_inputs
+
+    @classmethod
+    def from_config(cls, config, vision_model_name_or_path):
+        return cls(config, vision_model_name_or_path)
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path: Optional[Union[str, os.PathLike]] = None,
+        *model_args,
+        **kwargs,
+    ) -> "HCXVisionForCausalLM":
+        assert pretrained_model_name_or_path is not None
+
+        save_only_vision = kwargs.pop("save_only_vision") if "save_only_vision" in kwargs else False
+        save_only_qformer = kwargs.pop("save_only_qformer") if "save_only_qformer" in kwargs else False
+        save_shard_size = kwargs.pop("save_shard_size") if "save_shard_size" in kwargs else "5GB"
+
+        if pretrained_model_name_or_path is not None:  # when evaluate or load instruction tunned model
+            model: HCXVisionForCausalLM = super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
+            model.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)
+
+        image_token_id = model.tokenizer.encode(IMAGE_LOC, add_special_tokens=False)
+        assert (
+            len(image_token_id) == 1
+        ), f'"<|dummy3|>" was not encoded into a single special token. Encoding result: {image_token_id}'
+        model.config.image_token_id = image_token_id[0]
+        
+        video_token_id = model.tokenizer.encode(VIDEO_LOC, add_special_tokens=False)
+        assert (
+            len(video_token_id) == 1
+        ), f'"<|_unuse_missing_100270|>" was not encoded into a single special token. Encoding result: {video_token_id}'
+        model.config.video_token_id = video_token_id[0]
+
+        model.save_only_vision = save_only_vision
+        model.save_only_qformer = save_only_qformer
+        model.save_shard_size = save_shard_size
+
+        return model
+
+    def get_language_model(self):
+        return self.language_model.base_model
+
+    def get_vision_model(self):
+        return self.vision_model
+
+    def save_pretrained(
+        self,
+        save_directory: Union[str, os.PathLike],
+        *args,
+        **kwargs,
+    ):
+        state_dict = kwargs["state_dict"] if "state_dict" in kwargs else self.state_dict()
+        partial_state_dict = self.get_pretrained_state_dict(
+            state_dict,
+            save_directory,
+        )
+        kwargs["state_dict"] = partial_state_dict
+        kwargs["safe_serialization"] = self.is_safetensor_save
+        kwargs.setdefault("max_shard_size", self.save_shard_size)
+        super().save_pretrained(save_directory, *args, **kwargs)
+
+    def get_pretrained_state_dict(self, state_dict, save_dir):
+        vision_key = "vision_model."
+        llm_keys = ["language_model."]
+        head_key = "lm_head."
+
+        for key in list(state_dict.keys()):
+            if self.save_only_vision:
+                for llm_key in llm_keys:
+                    if llm_key in key:
+                        state_dict.pop(key)
+                if key.startswith(head_key):
+                    state_dict.pop(key)
+
+                elif self.save_only_qformer:
+                    if f"{vision_key}" in key:
+                        state_dict.pop(key)
+
+        return state_dict
+
+
+
+class HCXVisionMlp(nn.Module):
+    def __init__(
+        self,
+        mm_projector_type,
+        in_features,
+        hidden_features=None,
+        out_features=None,
+        act_layer=nn.GELU,
+    ):
+        super().__init__()
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+        self.mm_projector_type = mm_projector_type
+        if self.mm_projector_type == "mlp":
+            self.fc1 = nn.Linear(in_features, hidden_features)
+            self.act = act_layer()
+            self.fc2 = nn.Linear(hidden_features, out_features)
+        elif self.mm_projector_type == "inverted_mlp":
+            self.fc1 = nn.Linear(in_features, 2 * hidden_features)
+            self.act = act_layer()
+            self.fc2 = nn.Linear(2 * hidden_features, out_features)
+        else:
+            raise NotImplementedError("{} is not implemented".format(self.mm_projector_type))
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.act(x)
+        x = self.fc2(x)
+        return x
+
+
+class HCXVisionCAbstractor(nn.Module):
+    """
+    This module is based on C-Abstractor, whose license is under apache-2.0.
+    You can check the original code at https://github.com/khanrc/honeybee/blob/main/honeybee/projectors/projectors.py
+    and we made necessary modifications.
+    """
+
+    def __init__(
+        self,
+        num_queries: int,
+        num_input_tokens: int,
+        encoder_hidden_size: int,
+        hidden_size: int,
+        output_hidden_size: int,
+        pos_emb: bool = True,
+        prenorm: bool = False,
+    ):
+        super().__init__()
+        self.num_input_tokens = num_input_tokens
+        self.output_hidden_size = output_hidden_size
+
+        # Positional embedding
+        if pos_emb:
+            self.pos_emb = torch.nn.Parameter(torch.zeros(1, num_input_tokens, encoder_hidden_size))
+            self.pos_emb.data.normal_(mean=0.0, std=0.02)
+        else:
+            self.pos_emb = None
+
+        # (Optional) Pre-normalization layer
+        if prenorm:
+            self.prenorm = LayerNorm(encoder_hidden_size)
+        else:
+            self.prenorm = None
+
+        self.build_net(num_queries, encoder_hidden_size, hidden_size, output_hidden_size)
+        self.dtype = next(self.parameters()).dtype
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> torch.Tensor:
+        """
+        Args:
+            x: (B, L, encoder_hidden_size) tensor from the visual backbone (e.g. CLIP visual encoder), including cls token.
+        """
+        if self.prenorm is not None:
+            x = self.prenorm(x)
+
+        if self.pos_emb is not None:
+            x = x + self.pos_emb
+
+        x = self._forward(
+            x,
+            num_queries_vis_abstractors=num_queries_vis_abstractors,
+            num_grids=num_grids,
+        )  # (B, L, output_hidden_size)
+
+        return x
+
+    def _forward(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> torch.Tensor:
+        # x: [B, L, dim]
+        B, L, dim = x.shape
+        hw = int(L**0.5)
+        x = rearrange(x, "b (h w) d -> b d h w", h=hw, w=hw)
+
+        if num_queries_vis_abstractors is not None:
+            assert num_grids is not None
+            return self._forward_adaptive_num_query(x, num_queries_vis_abstractors, num_grids)
+
+        x = self.net(x)
+        x = rearrange(x, "b d h w -> b (h w) d")
+        x = self.readout(x)
+        return x
+
+    def _forward_adaptive_num_query(
+        self,
+        x: torch.Tensor,
+        num_queries_vis_abstractors: Optional[List[List[int]]] = None,
+        num_grids: Optional[List[int]] = None,
+    ) -> List[torch.Tensor]:
+        # self.net is consisted by 3 layers (s1, sampler, s2)
+        assert len(self.net) == 3
+
+        x = self.net[0](x)  # s1
+        new_x = []
+        for i, num_queries in enumerate(num_queries_vis_abstractors):
+            hw = int(num_queries**0.5)
+            sampler = nn.AdaptiveAvgPool2d((hw, hw))
+            out = sampler(x[num_grids[i] : num_grids[i + 1], :])
+            out = self.net[2](out)  # s2
+
+            out = rearrange(out, "b d h w -> b (h w) d")
+            out = self.readout(out)
+
+            new_x.append(out)
+        return new_x
+
+    def build_net(
+        self,
+        n_queries: int,
+        encoder_hidden_size: int,
+        hidden_size: int,
+        output_hidden_size: int,
+        depth: int = 3,
+        mlp_depth: int = 2,
+    ):
+        assert (n_queries**0.5).is_integer(), f"n_queries must be square number. n_queries: {n_queries}"
+        hw = int(n_queries**0.5)
+
+        # RegBlock = ResBlock + SE
+        RegBlock = partial(
+            RegStage,
+            stride=1,
+            dilation=1,
+            act_layer=nn.SiLU,
+            norm_layer=LayerNorm2d,
+        )
+
+        s1 = RegBlock(
+            depth,
+            encoder_hidden_size,
+            hidden_size,
+        )
+        sampler = nn.AdaptiveAvgPool2d((hw, hw))
+        s2 = RegBlock(
+            depth,
+            hidden_size,
+            hidden_size,
+        )
+
+        self.net = nn.Sequential(s1, sampler, s2)
+        self.readout = self.build_mlp(mlp_depth, hidden_size, output_hidden_size)
+
+    def build_mlp(
+        self,
+        depth: int,
+        hidden_size: int,
+        output_hidden_size: int,
+    ):
+        layers = [nn.Linear(hidden_size, output_hidden_size)]
+        for _ in range(1, depth):
+            layers.append(nn.SiLU())
+            layers.append(nn.Linear(output_hidden_size, output_hidden_size))
+        return nn.Sequential(*layers)
+

preprocessor_config.json

@@ -0,0 +1,135 @@
+{
+  "anyres": true,
+  "auto_map": {
+    "AutoImageProcessor": "image_processing_hyperclovax.HCXImageProcessor",
+    "AutoProcessor": "processing_hyperclovax.HCXProcessor"
+  },
+  "crop_size": {
+    "height": 378,
+    "width": 378
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_class": "AutoImageProcessor",
+  "image_processor_type": "HCXImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "num_queries_vis_abstractor_image": 81,
+  "num_queries_vis_abstractor_video_slow": 81,
+  "num_queries_vis_abstractor_video_fast": 9,
+  "first_last_frames_slow_video": false,
+  "pad_to_square": true,
+  "patch_size": 14,
+  "possible_resolutions": [
+    [
+      378,
+      378
+    ],
+    [
+      378,
+      756
+    ],
+    [
+      378,
+      1134
+    ],
+    [
+      378,
+      1512
+    ],
+    [
+      378,
+      1890
+    ],
+    [
+      378,
+      2268
+    ],
+    [
+      378,
+      2646
+    ],
+    [
+      378,
+      3024
+    ],
+    [
+      378,
+      3402
+    ],
+    [
+      756,
+      378
+    ],
+    [
+      756,
+      756
+    ],
+    [
+      756,
+      1134
+    ],
+    [
+      756,
+      1512
+    ],
+    [
+      1134,
+      378
+    ],
+    [
+      1134,
+      756
+    ],
+    [
+      1134,
+      1134
+    ],
+    [
+      1512,
+      378
+    ],
+    [
+      1512,
+      756
+    ],
+    [
+      1890,
+      378
+    ],
+    [
+      2268,
+      378
+    ],
+    [
+      2646,
+      378
+    ],
+    [
+      3024,
+      378
+    ],
+    [
+      3402,
+      378
+    ]
+  ],
+  "processor_class": "HCXProcessor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 378
+  },
+  "unpad": true
+}

processing_hyperclovax.py

@@ -0,0 +1,912 @@
+import copy
+import os
+import re
+import uuid
+from typing import Dict, List, Optional, Union
+
+import numpy as np
+import PIL
+from PIL import Image
+import torch
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ImageInput, load_image
+from transformers.processing_utils import (
+    AllKwargsForChatTemplate,
+    ChatTemplateLoadKwargs,
+    ProcessingKwargs,
+    ProcessorMixin,
+    Unpack,
+)
+from transformers.tokenization_utils_base import AudioInput, TextInput
+from transformers.utils import (
+    is_torch_device,
+    is_torch_dtype,
+    logging,
+    requires_backends,
+)
+from transformers.utils.chat_template_utils import render_jinja_template
+from transformers.video_utils import VideoInput, VideoMetadata, load_video
+
+logger = logging.get_logger(__name__)
+
+
+class HCXBatchFeature(BatchFeature):
+    def to(self, *args, **kwargs) -> "BatchFeature":
+        """
+        Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in
+        different `dtypes` and sending the `BatchFeature` to a different `device`.
+
+        Args:
+            args (`Tuple`):
+                Will be passed to the `to(...)` function of the tensors.
+            kwargs (`Dict`, *optional*):
+                Will be passed to the `to(...)` function of the tensors.
+                To enable asynchronous data transfer, set the `non_blocking` flag in `kwargs` (defaults to `False`).
+
+        Returns:
+            [`BatchFeature`]: The same instance after modification.
+        """
+        requires_backends(self, ["torch"])
+        import torch  # noqa
+
+        new_data = {}
+        device = kwargs.get("device")
+        non_blocking = kwargs.get("non_blocking", False)
+        # Check if the args are a device or a dtype
+        if device is None and len(args) > 0:
+            # device should be always the first argument
+            arg = args[0]
+            if is_torch_dtype(arg):
+                # The first argument is a dtype
+                pass
+            elif isinstance(arg, str) or is_torch_device(arg) or isinstance(arg, int):
+                device = arg
+            else:
+                # it's something else
+                raise ValueError(f"Attempting to cast a BatchFeature to type {str(arg)}. This is not supported.")
+        # We cast only floating point tensors to avoid issues with tokenizers casting `LongTensor` to `FloatTensor`
+        for k, v in self.items():
+            # check if v is a floating point
+            if isinstance(v, torch.Tensor) and torch.is_floating_point(v):
+                # cast and send to device
+                new_data[k] = v.to(*args, **kwargs)
+            elif isinstance(v, torch.Tensor) and device is not None:
+                new_data[k] = v.to(device=device, non_blocking=non_blocking)
+            elif "pixel_values" in k:
+                new_pixel_values_batch = []
+                for _v in v:
+                    pixel_values = [pixel_value.to(device=device, non_blocking=non_blocking) for pixel_value in _v]
+                    new_pixel_values_batch.append(pixel_values)
+                new_data[k] = new_pixel_values_batch
+            else:
+                new_data[k] = v
+        self.data = new_data
+        return self
+
+
+class HCXProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "text_kwargs": {
+            "return_tensors": "pt",
+            "calc_non_vision_query_lengths": False,
+        },
+        "images_kwargs": {},
+        "audio_kwargs": {},
+        "videos_kwargs": {
+            "max_image_cnt": 12,
+            "max_num_grids": 9,
+        },
+    }
+
+
+class HCXProcessor(ProcessorMixin):
+    attributes = ["image_processor", "tokenizer"]
+    valid_kwargs = ["chat_template"]
+
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = ("GPT2Tokenizer", "GPT2TokenizerFast")
+
+    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
+        self.image_token = "<|dummy3|>"
+        self.video_token = "<|_unuse_missing_100270|>"
+        self.image_token_pattern = re.compile(r"<\|dummy3\|>")
+        self.video_token_pattern = re.compile(r"<\|_unuse_missing_100270\|>")
+        self.image_video_token_pattern = re.compile(r"<\|dummy3\|>|<\|_unuse_missing_100270\|>")
+        self.image_token_id = (
+            tokenizer.image_token_id
+            if getattr(tokenizer, "image_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.image_token)
+        )
+        self.video_token_id = (
+            tokenizer.video_token_id
+            if getattr(tokenizer, "video_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.video_token)
+        )
+        super().__init__(image_processor, tokenizer, chat_template=chat_template)
+
+    def apply_chat_template(
+        self,
+        conversation: Union[list[dict[str, str]], list[list[dict[str, str]]]],
+        chat_template: Optional[str] = None,
+        **kwargs: Unpack[AllKwargsForChatTemplate],
+    ) -> str:
+        """
+        Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input
+        conversations to turn them into a single tokenizable string.
+
+        The input is expected to be in the following format, where each message content is a list consisting of text and
+        optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form
+        `pixel_values` when `return_dict=True`. If not provided, one will get only the formatted text, optionally tokenized text.
+
+        conversation = [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "image", "image": "https://www.ilankelman.org/stopsigns/australia.jpg"},
+                    {"type": "text", "text": "Please describe this image in detail."},
+                ],
+            },
+        ]
+
+        Args:
+            conversation (`Union[List[Dict, [str, str]], List[List[Dict[str, str]]]]`):
+                The conversation to format.
+            chat_template (`Optional[str]`, *optional*):
+                The Jinja template to use for formatting the conversation. If not provided, the tokenizer's
+                chat template is used.
+        """
+
+        if chat_template is None:
+            if isinstance(self.chat_template, dict) and "default" in self.chat_template:
+                chat_template = self.chat_template["default"]
+            elif isinstance(self.chat_template, dict):
+                raise ValueError(
+                    'The processor has multiple chat templates but none of them are named "default". You need to specify'
+                    " which one to use by passing the `chat_template` argument. Available templates are: "
+                    f"{', '.join(self.chat_template.keys())}"
+                )
+            elif self.chat_template is not None:
+                chat_template = self.chat_template
+            else:
+                raise ValueError(
+                    "Cannot use apply_chat_template because this processor does not have a chat template."
+                )
+        else:
+            if isinstance(self.chat_template, dict) and chat_template in self.chat_template:
+                # It's the name of a template, not a full template string
+                chat_template = self.chat_template[chat_template]
+            else:
+                # It's a template string, render it directly
+                chat_template = chat_template
+
+        if kwargs.get("continue_final_message", False):
+            if kwargs.get("add_generation_prompt", False):
+                raise ValueError(
+                    "continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead."
+                )
+            if kwargs.get("return_assistant_tokens_mask", False):
+                raise ValueError("continue_final_message is not compatible with return_assistant_tokens_mask.")
+
+        # Fill sets of kwargs that should be used by different parts of template
+        processed_kwargs = {
+            "mm_load_kwargs": {},
+            "template_kwargs": {},
+        }
+
+        for kwarg_type in processed_kwargs:
+            for key in AllKwargsForChatTemplate.__annotations__[kwarg_type].__annotations__.keys():
+                kwarg_type_defaults = AllKwargsForChatTemplate.__annotations__[kwarg_type]
+                default_value = getattr(kwarg_type_defaults, key, None)
+                value = kwargs.pop(key, default_value)
+                if value is not None and not isinstance(value, dict):
+                    processed_kwargs[kwarg_type][key] = value
+
+        # Pass unprocessed custom kwargs
+        processed_kwargs["template_kwargs"].update(kwargs)
+
+        if isinstance(conversation, (list, tuple)) and (
+            isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "content")
+        ):
+            is_batched = True
+            conversations = conversation
+        else:
+            is_batched = False
+            conversations = [conversation]
+
+        tokenize = processed_kwargs["template_kwargs"].pop("tokenize", False)
+        return_dict = processed_kwargs["template_kwargs"].pop("return_dict", False)
+        mm_load_kwargs = processed_kwargs["mm_load_kwargs"]
+
+        if tokenize:
+            batch_images, batch_videos = [], []
+            batch_audios = []
+            batch_video_metadata = []
+            for conversation in conversations:
+                images, videos = [], []
+                video_metadata = []
+                for message in conversation:
+                    visuals = [content for content in message["content"] if content["type"] in ["image", "video"]]
+                    audio_fnames = [
+                        content[key]
+                        for content in message["content"]
+                        for key in ["audio", "url", "path"]
+                        if key in content and content["type"] == "audio"
+                    ]
+                    image_fnames = [
+                        vision_info[key]
+                        for vision_info in visuals
+                        for key in ["image", "url", "path", "base64"]
+                        if key in vision_info and vision_info["type"] == "image"
+                    ]
+                    video_fnames = [
+                        vision_info[key]
+                        for vision_info in visuals
+                        for key in ["video", "url", "path"]
+                        if key in vision_info and vision_info["type"] == "video"
+                    ]
+
+                    for fname in image_fnames:
+                        images.append(load_image(fname))
+
+                    # Audio models do not accept nested list of audios (yet!) so we construct a flat input audio list
+                    if not mm_load_kwargs["load_audio_from_video"]:
+                        for fname in audio_fnames:
+                            batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
+                    else:
+                        for fname in video_fnames:
+                            batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
+
+                    for fname in video_fnames:
+                        if isinstance(fname, (list, tuple)) and isinstance(fname[0], str):
+                            video = [np.array(load_image(image_fname)) for image_fname in fname]
+                            # create a 4D video because `load_video` always returns a 4D array
+                            video = np.stack(video)
+                            metadata = None
+                            logger.warning(
+                                "When loading the video from list of images, we cannot infer metadata such as `fps` or `duration`. "
+                                "If your model uses this metadata during processing, please load the whole video and let the model sample frames instead."
+                            )
+                        else:
+                            # TODO: raushan, should be `self.video_processor.load_video_for_model` when API is added
+                            video, metadata = self._load_video_for_model(
+                                fname,
+                                num_frames=mm_load_kwargs.get("num_frames", None),
+                                fps=mm_load_kwargs.get("video_fps", None),
+                                backend=mm_load_kwargs["video_load_backend"],
+                                **kwargs,
+                            )
+                        videos.append(video)
+                        video_metadata.append(metadata)
+
+                # Currently all processors can accept nested list of batches, but not flat list of visuals
+                # So we'll make a batched list of images and let the processor handle it
+                if images:
+                    batch_images.append(images)
+                if videos:
+                    batch_videos.append(videos)
+                    batch_video_metadata.append(video_metadata)
+
+            # Process conversation with video/image information if needed. Then convert into a prompt using Jinja template
+            conversations = self._process_messages_for_chat_template(
+                conversations,
+                batch_images=batch_images,
+                batch_videos=batch_videos,
+                batch_video_metadata=batch_video_metadata,
+                **processed_kwargs["mm_load_kwargs"],
+            )
+
+        prompt, generation_indices = render_jinja_template(
+            conversations=conversations,
+            chat_template=chat_template,
+            **processed_kwargs["template_kwargs"],  # different flags such as `return_assistant_mask`
+            **self.tokenizer.special_tokens_map,  # tokenizer special tokens are used by some templates
+        )
+
+        if not is_batched:
+            prompt = prompt[0]
+
+        if tokenize:
+            # Tokenizer's `apply_chat_template` never adds special tokens when tokenizing
+            # But processor's `apply_chat_template` didn't have an option to tokenize, so users had to format the prompt
+            # and pass it to the processor. Users thus never worried about special tokens relying on processor handling
+            # everything internally. The below line is to keep BC for that and be able to work with model that have
+            # special tokens in the template (consistent with tokenizers). We dont want to raise warning, it will flood command line
+            # without actionable solution for users
+            single_prompt = prompt[0] if is_batched else prompt
+            if self.tokenizer.bos_token is not None and single_prompt.startswith(self.tokenizer.bos_token):
+                kwargs["add_special_tokens"] = False
+
+            out = self(
+                text=prompt,
+                images=batch_images if batch_images else None,
+                videos=batch_videos if batch_videos else None,
+                audio=batch_audios if batch_audios else None,
+                **kwargs,
+            )
+            if return_dict:
+                if processed_kwargs["template_kwargs"].get("return_assistant_tokens_mask", False):
+                    assistant_masks = []
+                    input_ids = out["input_ids"]
+                    for i in range(len(input_ids)):
+                        current_mask = [0] * len(input_ids[i])
+                        for assistant_start_char, assistant_end_char in generation_indices[i]:
+                            start_token = out.char_to_token(i, assistant_start_char)
+                            end_token = out.char_to_token(i, assistant_end_char - 1)
+                            if start_token is None:
+                                # start_token is out of bounds maybe due to truncation.
+                                break
+                            for token_id in range(start_token, end_token + 1 if end_token else len(input_ids[i])):
+                                current_mask[token_id] = 1
+                        assistant_masks.append(current_mask)
+                    out["assistant_masks"] = assistant_masks
+                    out.convert_to_tensors(tensor_type=kwargs.get("return_tensors", None))
+                
+                # vllm needs vision_query_lengths, but hf model doesn't need it
+                del out["vision_query_lengths_images"]
+                del out["vision_query_lengths_videos"]
+                return out
+            else:
+                return out["input_ids"]
+    
+    def repeat_dummy_tokens(self, input_ids, target_token_id, vision_query_lengths):
+        input_ids = input_ids.clone().detach()
+        batch_indices, target_indices = torch.where(input_ids == target_token_id)
+        batch_size = input_ids.shape[0]
+
+        new_input_ids = [[] for _ in range(batch_size)]
+        start_indices = [0 for _ in range(batch_size)]
+        counter = [0 for _ in range(batch_size)]
+        for batch_idx, target_idx in zip(batch_indices, target_indices):
+            start_idx = start_indices[batch_idx]
+            new_input_ids[batch_idx].append(input_ids[batch_idx][start_idx:target_idx])
+            query_length = vision_query_lengths[batch_idx][counter[batch_idx]]
+            new_input_ids[batch_idx].append(input_ids[batch_idx][target_idx].repeat(query_length))
+            start_indices[batch_idx] = target_idx + 1
+            counter[batch_idx] += 1
+
+        for batch_idx in range(batch_size):
+            start_idx = start_indices[batch_idx]
+            new_input_ids[batch_idx].append(input_ids[batch_idx][start_idx:])  # append remaining tokens
+            new_input_ids[batch_idx] = torch.cat(new_input_ids[batch_idx], dim=0)
+
+        new_input_ids = torch.stack(new_input_ids)
+        return new_input_ids
+
+    def _load_video_for_model(
+        self,
+        video: str,
+        num_frames: Optional[int] = None,
+        fps: Optional[int] = None,
+        backend: str = "opencv",
+        **kwargs: Unpack[HCXProcessorKwargs],
+    ) -> List[ImageInput]:
+        """
+        Overrided function.
+
+        Loads `video` to a List[PIL.Image] (llava style)
+
+        Args:
+            video (`str`):
+                The video to convert to the numpy array format. Can be a link to video or local path.
+            num_frames (`int`, *optional*):
+                Number of frames to sample uniformly. If not passed, the whole video is loaded.
+            fps (`int`, *optional*):
+                Number of frames to sample per second. Should be passed only when `num_frames=None`.
+                If not specified and `num_frames==None`, all frames are sampled.
+            backend (`str`, *optional*, defaults to `"opencv"`):
+                The backend to use when loading the video. Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "opencv".
+
+        Returns:
+            Tuple[`np.array`, Dict]: A tuple containing:
+                - List[PIL.Image] of frames in RGB.
+                - Metadata dictionary.
+        """
+        output_kwargs = self._merge_kwargs(
+            HCXProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        logger.warning_once(f"num_frames control via argument is not supported yet. Ignored num_frames: {num_frames}.")
+        logger.warning_once(f"fps control via argument is not supported yet. Ignored fps: {fps}.")
+        logger.warning_once(f"backend control via argument is not supported yet. Ignored backend: {backend}.")
+
+        # video_loaded, video_metadata = load_video(
+        #     video, backend="decord", num_frames=32
+        # )
+        # frame_interval = int(video_metadata.total_num_frames / 32)
+        # time_interval = frame_interval / video_metadata.fps
+        # video_metadata.time_interval = time_interval
+        
+        def _hcx_sample_indices_fn(metadata: VideoMetadata, num_frames=None, fps=None, **kwargs):
+            max_num_grids = output_kwargs["videos_kwargs"]["max_num_grids"]
+            max_image_cnt = output_kwargs["videos_kwargs"]["max_image_cnt"]
+            frame_indices, time_interval = extract_frame_indices(
+                metadata.duration,
+                metadata.total_num_frames,
+                metadata.fps,
+                max_num_grids,
+                max_image_cnt,
+                default_interval=0.4,
+            )
+            metadata.time_interval = time_interval
+            return np.array(frame_indices)
+
+        video_loaded, video_metadata = None, None
+        for backend in ["decord", "pyav", "opencv", "torchvision"]:
+            try:
+                video_loaded, video_metadata = load_video(
+                    video, sample_indices_fn=_hcx_sample_indices_fn, backend=backend
+                )
+                break
+            except Exception as e:
+                logger.error(f"Error loading video with {backend} backend: {e}")
+                continue
+
+        assert video_loaded is not None, "Failed to load video with any backend"
+
+        return video_loaded, video_metadata
+
+    def _process_messages_for_chat_template(
+        self,
+        conversation: List[List[Dict[str, str]]],
+        batch_images: List[List[ImageInput]],
+        batch_videos: List[List[VideoInput]],
+        batch_video_metadata: List[List[Dict[str, any]]],
+        **mm_load_kwargs: Unpack[ChatTemplateLoadKwargs],
+    ):
+        """
+        Overrided function.
+        Used within `apply_chat_template` when a model has a special way to process conversation history. For example,
+        video models might want to specify in the prompt the duration of video or which frame indices at which timestamps
+        were sampled. This information cannot be accessed before the video is loaded.
+
+        For most models it is a no-op, and must be overridden by model processors which require special processing.
+
+        Args:
+            conversation (`List[Dict, str, str]`):
+                The conversation to process. Always comes in batched format.
+            batch_images (`List[List[ImageInput]]`):
+                Batch of images that were loaded from url/path defined in the conversation. The images
+                are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL` images
+                per batch.
+            batch_videos (`List[List[ImageInput]]`):
+                Batch of videos that were loaded from url/path defined in the conversation. The videos
+                are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL.Image`
+                per batch.
+            batch_video_metadata (`List[List[Dict[[str, any]]]]`):
+                Batch of metadata returned from loading videos. That includes video fps, duration and total number of framer in original video.
+                Metadata are ordered in the same way as `batch_videos`. Comes in nested list format, one list of `Dict`
+                per batch.
+        """
+
+        is_video_in_conversation = False
+        for batch_idx, messages in enumerate(conversation):
+            is_video_in_messages = False
+            is_image_in_messages = False
+            for message in messages:
+                for content in message["content"]:
+                    if content["type"] == "video":
+                        is_video_in_messages = True
+                    elif content["type"] == "image":
+                        is_image_in_messages = True
+            if not is_video_in_messages:
+                batch_videos.insert(batch_idx, [])
+                batch_video_metadata.insert(batch_idx, [])
+            if not is_image_in_messages:
+                batch_images.insert(batch_idx, [])
+
+            is_video_in_conversation = is_video_in_conversation or is_video_in_messages
+
+        if not is_video_in_conversation:
+            return conversation
+
+        # conversation processing
+        new_conversation = []
+        for batch_idx, messages in enumerate(conversation):
+            video_counter = 0
+            new_messages = []
+
+            for message in messages:
+                new_message = {
+                    "role": message["role"],
+                    "content": [],
+                }
+                for content in message["content"]:
+                    if content["type"] == "video":
+                        video = batch_videos[batch_idx][video_counter]
+                        video_meta = batch_video_metadata[batch_idx][video_counter]
+
+                        time_stamps = calc_timestamp_video_grids(video, video_meta.time_interval, max_grid_shape=(3, 3))
+                        video_counter += 1
+
+                        if "filename" in content:
+                            filename = content["filename"]
+                        else:
+                            filename = content["video"].split("/")[-1]
+                            if len(filename) > 50:
+                                filename = f"{uuid.uuid4().hex}.mp4"
+                        basename, ext = os.path.splitext(filename)
+                        if ext == "":
+                            ext = ".mp4"
+
+                        for frame_idx, time_stamp in enumerate(time_stamps):
+                            if frame_idx == len(video) - 1:
+                                # final_grid
+                                new_content = {
+                                    "filename": f"{basename}-{frame_idx}{ext}",
+                                    "video": content["video"],
+                                    "type": "video",
+                                    "video_time_stamp": time_stamp,
+                                    "lens_keywords": content["lens_keywords"],
+                                    "lens_local_keywords": content["lens_local_keywords"],
+                                    "speech_to_text": content["speech_to_text"],
+                                    "is_final_grid": True,
+                                }
+                                new_message["content"].append(new_content)
+                            else:
+                                new_content = {
+                                    "filename": f"{basename}-{frame_idx}{ext}",
+                                    "video": content["video"],
+                                    "type": "video",
+                                    "video_time_stamp": time_stamp,
+                                }
+                                new_message["content"].append(new_content)
+                    else:
+                        new_message["content"].append(copy.deepcopy(content))
+                new_messages.append(new_message)
+            new_conversation.append(new_messages)
+
+        return new_conversation
+
+    def __call__(
+        self,
+        text: TextInput = None,
+        images: List[List[ImageInput]] = None,
+        videos: List[List[VideoInput]] = None,
+        audio: AudioInput = None,
+        **kwargs: Unpack[HCXProcessorKwargs],
+    ):
+        output_kwargs = self._merge_kwargs(
+            HCXProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        # prepare model inputs
+        mm_inputs = {
+            "pixel_values_images": [],
+            "image_sizes_images": [],
+            "vision_query_lengths_images": [],
+            "pixel_values_videos": [],
+            # "image_sizes_videos": [],
+            "vision_query_lengths_videos": [],
+        }
+        calc_non_vision_query_lengths = output_kwargs["text_kwargs"].pop("calc_non_vision_query_lengths")
+        if calc_non_vision_query_lengths:
+            mm_inputs["non_vision_query_lengths"] = []
+
+        # video processing
+        if videos is not None:
+            vit_input_size = self.image_processor.crop_size["width"]
+
+            video_kwargs = copy.deepcopy(output_kwargs["videos_kwargs"])
+
+            for videos_in_single_conversation in videos:
+                pixel_values_videos = []
+                vision_query_lengths_videos = []
+
+                for video_frames in videos_in_single_conversation:
+                    if len(video_frames) == 0:
+                        mm_inputs["pixel_values_videos"].append([])
+                        mm_inputs["vision_query_lengths_videos"].append([])
+                        continue
+                    video_frames_combined = combine_frames_into_images(
+                        video_frames, max_grid_shape=(3, 3), vit_input_size=vit_input_size
+                    )
+                    video_kwargs["is_video"] = True
+                    video_kwargs["return_tensors"] = None
+
+                    frames_processed = self.image_processor(images=video_frames_combined, **video_kwargs)
+                    sizes = [(size["width"], size["height"]) for size in frames_processed["image_sizes"]]
+
+                    pixel_values_videos.extend(frames_processed["pixel_values"])
+                    vision_query_lengths_videos.extend(frames_processed["vision_query_lengths"])
+
+                mm_inputs["pixel_values_videos"].append(pixel_values_videos)
+                mm_inputs["vision_query_lengths_videos"].append(vision_query_lengths_videos)
+
+        # image processing
+        if images is not None:
+            image_kwargs = copy.deepcopy(output_kwargs["images_kwargs"])
+            image_kwargs["is_video"] = False
+            image_kwargs["return_tensors"] = None
+
+            for images_in_single_conversation in images:
+                if isinstance(images_in_single_conversation, PIL.Image.Image): # single item to batch
+                    images_in_single_conversation = [images_in_single_conversation, ]
+                if len(images_in_single_conversation) == 0:
+                    mm_inputs["pixel_values_images"].append([])
+                    mm_inputs["image_sizes_images"].append([])
+                    mm_inputs["vision_query_lengths_images"].append([])
+                    continue
+                images_processed = self.image_processor(images=images_in_single_conversation, **image_kwargs)
+                sizes = [(size["width"], size["height"]) for size in images_processed["image_sizes"]]
+
+                mm_inputs["pixel_values_images"].append(images_processed["pixel_values"])
+                mm_inputs["image_sizes_images"].append(sizes)
+                mm_inputs["vision_query_lengths_images"].append(images_processed["vision_query_lengths"])
+
+        # text processing
+        def _create_replacer(_target_token, _replacements):
+            _iterator = iter(_replacements)
+
+            def _replacer(match_obj):
+                # return self.image_token
+                num_query_tokens = next(_iterator)
+                return "".join([_target_token for _ in range(num_query_tokens)])
+            return _replacer
+
+        text_inputs = {}
+        if text is not None:
+            if not isinstance(text, list):
+                text = [text]
+
+            if images is not None:
+                new_texts = []
+                for batch_idx, text_in_single_conversation in enumerate(text):
+                    new_text = self.image_token_pattern.sub(
+                        _create_replacer(self.image_token, mm_inputs["vision_query_lengths_images"][batch_idx]),
+                        text_in_single_conversation,
+                    )
+                    new_texts.append(new_text)
+                text = new_texts
+
+            if videos is not None:
+                new_texts = []
+                for batch_idx, text_in_single_conversation in enumerate(text):
+                    new_text = self.video_token_pattern.sub(
+                        _create_replacer(self.video_token, mm_inputs["vision_query_lengths_videos"][batch_idx]),
+                        text_in_single_conversation,
+                    )
+                    new_texts.append(new_text)
+                text = new_texts
+
+            text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+
+        # audio processing
+        if audio is not None:
+            raise NotImplementedError("Audio processing is not supported yet.")
+
+        return HCXBatchFeature(data={**text_inputs, **mm_inputs})
+
+    def decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to Siglip2Tokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
+        the docstring of this method for more information.
+        """
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to Siglip2Tokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
+        refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def post_process_image_text_to_text(
+        self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
+    ):
+        """
+        Post-process the output of the model to decode the text.
+
+        Args:
+            generated_outputs (`torch.Tensor` or `np.ndarray`):
+                The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
+                or `(sequence_length,)`.
+            skip_special_tokens (`bool`, *optional*, defaults to `True`):
+                Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
+            Clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
+                Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method.
+            **kwargs:
+                Additional arguments to be passed to the tokenizer's `batch_decode method`.
+
+        Returns:
+            `List[str]`: The decoded text.
+        """
+        return self.tokenizer.batch_decode(
+            generated_outputs,
+            skip_special_tokens=skip_special_tokens,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+    @property
+    def model_input_names(self):
+        tokenizer_input_names = self.tokenizer.model_input_names
+        image_processor_input_names = self.image_processor.model_input_names
+        names_from_processor = list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
+        return names_from_processor + []
+
+
+def extract_frame_indices(play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=0.4):
+    """
+    Extracts specific frame indices from a video based on duration, frame count, and sampling strategy.
+
+    The function determines which frames to extract given the video duration (`play_time`),
+    total frame count, and frame rate. It samples frames at regular intervals (default: 0.4s),
+    but if the number of frames exceeds the limit defined by `max_num_grids * max_image_cnt`,
+    it performs uniform sampling to stay within that limit.
+
+    Args:
+        play_time (float): Total play time of the video in seconds.
+        total_frames (int): Total number of frames in the video.
+        fps (float): Frames per second of the video.
+        max_num_grids (int): Maximum number of grids to display.
+        max_image_cnt (int): Maximum number of images per grid.
+        default_interval (float, optional): Interval in seconds between frame samples. Defaults to 0.4.
+
+    Returns:
+        Tuple:
+            frame_indices (List[int]): A list of selected frame indices.
+            time_interval (float): Time interval between selected frames (in seconds).
+    """
+
+    # Calculate how many frames to extract with the default interval
+    default_frame_count = int(play_time / default_interval)
+
+    # Maximum frames allowed based on max_num_grids and max_image_cnt
+    max_frames_allowed = max_num_grids * max_image_cnt
+
+    # Determine whether we can use the default interval or need uniform sampling
+    if default_frame_count <= max_frames_allowed:
+        # Default interval is sufficient, extract frames every 0.4 seconds
+        frame_interval = int(total_frames / default_frame_count)
+    else:
+        # Use uniform sampling to fit within max_frames_allowed
+        frame_interval = int(total_frames / max_frames_allowed)
+
+    # Extract frame indices at the calculated interval
+    selected_indices = list(range(0, total_frames, frame_interval))
+
+    time_interval = frame_interval / fps
+
+    # Ensure the number of selected indices does not exceed max_frames_allowed
+    return selected_indices[:max_frames_allowed], time_interval
+
+
+def calc_timestamp_video_grids(frames, time_interval, max_grid_shape=(3, 3)):
+    """
+    Calculates the time range labels for each grid in a video.
+
+    Args:
+        frames (List[PIL.Image.Image]): A list of frames extracted from a video.
+        time_interval (float): Time interval (in seconds) between consecutive frames.
+        max_grid_shape (Tuple[int, int], optional): The maximum grid shape as (rows, cols). Defaults to (3, 3).
+        vit_input_size (int, optional): The target size (height and width) for the Vision Transformer input. Defaults to 378.
+
+    Returns:
+        Tuple:
+            image_time_stamps (List[str]): A list of time span labels for each combined image,
+                e.g., ["0.00s~1.50s", "1.50s~3.00s", ...].
+    """
+    max_num_grids = max_grid_shape[0] * max_grid_shape[1]
+    # assert (
+    #     max_grid_shape[1] == 1
+    # ), f"For video processing, decided to concatenate frames horizontally into a wide image."
+
+    # Calculate the number of canvases needed.
+    num_frames = len(frames)
+    num_canvases = num_frames // max_num_grids
+    leftover_frames = num_frames % max_num_grids
+
+    time_stamp = 0  # second
+    image_time_stamps = []
+
+    for canvas_idx in range(num_canvases):
+        # Determine the frames to fill in the current canvas.
+        start_idx = canvas_idx * max_num_grids
+        end_idx = min(start_idx + max_num_grids, num_frames)
+
+        # Append the current canvas to the result list.
+        frame_cnt = end_idx - start_idx
+        image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
+        time_stamp += frame_cnt * time_interval
+
+    if leftover_frames > 0:
+        # Add the current canvas to the list of combined images.
+        frame_cnt = leftover_frames
+        image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
+        time_stamp += frame_cnt * time_interval
+
+    return image_time_stamps
+
+
+def combine_frames_into_images(frames, max_grid_shape=(3, 3), vit_input_size=378):
+    """
+    Combines a sequence of video frames into grid-based images and generates corresponding time range labels.
+
+    Frames are grouped and arranged into a grid (e.g., 3x3) such that each combined image contains up to
+    `max_grid_shape[0] * max_grid_shape[1]` frames. Each combined image is resized to the given ViT input size.
+
+    Args:
+        frames (NDArray): (num_frames, H, W, C) shape. A list of frames extracted from a video.
+        time_interval (float): Time interval (in seconds) between consecutive frames.
+        max_grid_shape (Tuple[int, int], optional): The maximum grid shape as (rows, cols). Defaults to (3, 3).
+        vit_input_size (int, optional): The target size (height and width) for the Vision Transformer input. Defaults to 378.
+
+    Returns:
+        Tuple:
+            image_list (List[PIL.Image.Image]): A list of grid-combined images.
+    """
+    max_num_grids = max_grid_shape[0] * max_grid_shape[1]
+    # assert (
+    #     max_grid_shape[1] == 1
+    # ), f"For video processing, decided to concatenate frames horizontally into a wide image."
+
+    # List to store the resulting combined images.
+    image_list = []
+
+    # Calculate the number of canvases needed.
+    num_frames = len(frames)
+    num_canvases = num_frames // max_num_grids
+    leftover_frames = num_frames % max_num_grids
+
+    # change frames (4d numpy tensor) to List[PIL.Image.Image]
+    frames = [Image.fromarray(frame) for frame in frames]
+
+    for canvas_idx in range(num_canvases):
+        # Initialize the current canvas.
+        combined_image = Image.new(
+            "RGB", (vit_input_size * max_grid_shape[0], vit_input_size * max_grid_shape[1]), color=(0, 0, 0)
+        )
+
+        # Determine the frames to fill in the current canvas.
+        start_idx = canvas_idx * max_num_grids
+        end_idx = min(start_idx + max_num_grids, num_frames)
+
+        for idx in range(start_idx, end_idx):
+            img = frames[idx]
+
+            # Resize each frame to a square shape.
+            img_resized = img.resize((vit_input_size, vit_input_size))
+
+            # Calculate the (row, column) position to place the frame within the grid layout.
+            local_idx = idx - start_idx
+            x_offset = (local_idx % max_grid_shape[0]) * vit_input_size
+            y_offset = (local_idx // max_grid_shape[0]) * vit_input_size
+
+            # Calculate the position to place the frame in the grid.
+            combined_image.paste(img_resized, (x_offset, y_offset))
+
+        # Append the current canvas to the result list.
+        image_list.append(combined_image)
+
+    if leftover_frames > 0:
+        # canvas_idx might be undefined; default to 0 if not previously assigned to avoid "referenced before assignment" error.
+        canvas_idx = num_canvases
+        # Add the remaining frames to the final canvas.
+        # combined_image = Image.new("RGB", (vit_input_size * leftover_frames, vit_input_size * 1), color=(0, 0, 0))  # hsk
+        combined_image = Image.new(
+            "RGB", (vit_input_size * max_grid_shape[0], vit_input_size * max_grid_shape[1]), color=(0, 0, 0)
+        )
+
+        for idx in range(leftover_frames):
+            img = frames[num_canvases * max_num_grids + idx]
+
+            # Resize the frame to a square (equal width and height).
+            img_resized = img.resize((vit_input_size, vit_input_size))
+
+            # Calculate the (row, column) position to place the frame within the grid layout.
+            # x_offset = (idx % leftover_frames) * vit_input_size  # hsk
+            # y_offset = (idx // leftover_frames) * vit_input_size  # hsk
+            x_offset = (idx % max_grid_shape[0]) * vit_input_size
+            y_offset = (idx // max_grid_shape[0]) * vit_input_size
+
+            # Calculate the position to place the frame within the grid layout.
+            combined_image.paste(img_resized, (x_offset, y_offset))
+
+        # Add the current canvas to the list of combined images.
+        image_list.append(combined_image)
+
+    return image_list

processor_config.json

@@ -0,0 +1,6 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_hyperclovax.HCXProcessor"
+  },
+  "processor_class": "HCXProcessor"
+}

special_tokens_map.json

@@ -0,0 +1,86 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|endofprompt|>",
+    "<|_unuse_missing_100256|>",
+    "<|_unuse_missing_100261|>",
+    "<|_unuse_missing_100262|>",
+    "<|_unuse_missing_100263|>",
+    "<|_unuse_missing_100264|>",
+    "<|_unuse_missing_100265|>",
+    "<|_unuse_missing_100266|>",
+    "<|_unuse_missing_100267|>",
+    "<|_unuse_missing_100268|>",
+    "<|_unuse_missing_100269|>",
+    "<|_unuse_missing_100270|>",
+    "<|dummy3|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|stop|>",
+    "<|endofturn|>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endofturn|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,507 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|_unuse_missing_100256|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100257": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100261": {
+      "content": "<|_unuse_missing_100261|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100262": {
+      "content": "<|_unuse_missing_100262|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100263": {
+      "content": "<|_unuse_missing_100263|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100264": {
+      "content": "<|_unuse_missing_100264|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|_unuse_missing_100265|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|_unuse_missing_100266|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100267": {
+      "content": "<|_unuse_missing_100267|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100268": {
+      "content": "<|_unuse_missing_100268|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100269": {
+      "content": "<|_unuse_missing_100269|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100270": {
+      "content": "<|_unuse_missing_100270|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100271": {
+      "content": "<|dummy3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100272": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100273": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100274": {
+      "content": "<|stop|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100275": {
+      "content": "<|endofturn|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100276": {
+      "content": "<|endofprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110491": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110492": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110493": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110494": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110495": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110496": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110497": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110498": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110499": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110500": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110501": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110502": {
+      "content": "<code_to_intermediate>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110503": {
+      "content": "<intermediate_to_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110504": {
+      "content": "<pr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110505": {
+      "content": "<pr_status>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110506": {
+      "content": "<pr_is_merged>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110507": {
+      "content": "<pr_base>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110508": {
+      "content": "<pr_file>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110509": {
+      "content": "<pr_base_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110510": {
+      "content": "<pr_diff>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110511": {
+      "content": "<pr_diff_hunk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110512": {
+      "content": "<pr_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110513": {
+      "content": "<pr_event_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110514": {
+      "content": "<pr_review>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110515": {
+      "content": "<pr_review_state>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110516": {
+      "content": "<pr_review_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110517": {
+      "content": "<pr_in_reply_to_review_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110518": {
+      "content": "<pr_in_reply_to_comment_id>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110519": {
+      "content": "<pr_diff_hunk_comment_line>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110520": {
+      "content": "<NAME>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110521": {
+      "content": "<EMAIL>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110522": {
+      "content": "<KEY>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110523": {
+      "content": "<PASSWORD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|endofprompt|>",
+    "<|_unuse_missing_100256|>",
+    "<|_unuse_missing_100261|>",
+    "<|_unuse_missing_100262|>",
+    "<|_unuse_missing_100263|>",
+    "<|_unuse_missing_100264|>",
+    "<|_unuse_missing_100265|>",
+    "<|_unuse_missing_100266|>",
+    "<|_unuse_missing_100267|>",
+    "<|_unuse_missing_100268|>",
+    "<|_unuse_missing_100269|>",
+    "<|_unuse_missing_100270|>",
+    "<|dummy3|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|stop|>",
+    "<|endofturn|>",
+    "<repo_name>",
+    "<file_sep>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>",
+    "<code_to_intermediate>",
+    "<intermediate_to_code>",
+    "<pr>",
+    "<pr_status>",
+    "<pr_is_merged>",
+    "<pr_base>",
+    "<pr_file>",
+    "<pr_base_code>",
+    "<pr_diff>",
+    "<pr_diff_hunk>",
+    "<pr_comment>",
+    "<pr_event_id>",
+    "<pr_review>",
+    "<pr_review_state>",
+    "<pr_review_comment>",
+    "<pr_in_reply_to_review_id>",
+    "<pr_in_reply_to_comment_id>",
+    "<pr_diff_hunk_comment_line>",
+    "<NAME>",
+    "<EMAIL>",
+    "<KEY>",
+    "<PASSWORD>"
+  ],
+  "auto_map": {
+    "AutoProcessor": "processing_hyperclovax.HCXProcessor"
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endofturn|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "HCXProcessor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}