init model

MODEL_LICENSE.md

@@ -0,0 +1,47 @@
+# CodeFuse COMMUNITY LICENSE AGREEMENT
+CodeFuse Release Date: September 8, 2023
+
+By clicking to agree or by using or distributing any portion or element of the Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
+
+1. Definitions.
+    a. This CodeFuse COMMUNITY LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
+    b. "Ant" or "We" (or "Us") shall mean Ant Group.
+    c. "CodeFuse" shall mean the large language models (including CodeFuse-13B and CodeFuse-CodeLlaMa-34B), and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, and other elements of the foregoing distributed by Us.
+    d. "Documentation" shall mean the specifications, manuals and documentation accompanying CodeFuse distributed by Us.
+    e. "Materials" shall mean, collectively, Ant's proprietary CodeFuse and Documentation (and any portion thereof) made available under this Agreement.
+    f. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+    g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
+    h. "Third Parties" (or "Third Party") shall mean individuals or legal entities that are not controlling, controlled by Us or You, or under common control with Us or You.
+    i. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
+
+2. Grant of Rights.
+You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Ant's intellectual property or other rights owned by Ant embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials.
+
+3. Redistribution.
+You may distribute or make the Materials or derivative works thereof available to a Third Party in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+    a. You shall provide a copy of this Agreement to such Third Party;
+    b. if You modify the CodeFuse model, You shall provide a prominent notice, stating how You have modified the CodeFuse model, to such Third Party; and 
+    c. You shall retain in all copies of the Materials that You distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "CodeFuse is licensed under the CodeFuse COMMUNITY LICENSE AGREEMENT, Copyright (c) Ant Group. All Rights Reserved." 
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such derivative works as a whole, provided Your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
+
+4. Rules of Use. 
+    You shall comply with applicable laws and regulations (including without limitation export controls or restrictions) in Your use of the Materials.
+   
+5. Intellectual Property. 
+    a. Ant retains ownership of all intellectual property rights in and to the Materials and derivatives made by or for Ant. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by You, You are and will be the owner of such derivative works and modifications.
+    b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of Ant, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials. 
+    c. If You commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Ant or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by You, then all licences granted to You under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought. 
+
+6. Disclaimer of Warranty and Limitation of Liability. 
+    a. Ant is not obligated to support, update, provide training for, or develop any further version of the Materials or to grant any license thereto.
+   b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM. 
+    c. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE MATERIALS AND ANY OUTPUT AND RESULTS. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT OR ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, FOR ANY DIRECT, OR INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, NO MATTER HOW IT'S CAUSED OR EVEN IF ANT OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+    d. You will defend, indemnify and hold harmless Ant from and against any claim by any Third Party arising out of or related to Your use or distribution of the Materials.
+
+7. Survival and Termination.
+    a. The term of this Agreement shall commence upon Your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
+    b. We may terminate this Agreement if You breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, You must delete and cease use of the Materials. Sections 6 and 8 shall survive the termination of this Agreement.
+
+8. Governing Law and Jurisdiction.
+    a. This Agreement and any dispute arising out of or relating to it, whether in contract, tort, negligence, products liability, or otherwise, will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
+    b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.

README.md

@@ -1,3 +1,255 @@
 ---
 license: apache-2.0
+tasks: 
+- code-generation
 ---
+# Model Card for CodeFuse-CodeLlama-34B
+<p align="center">
+    <img src="https://modelscope.cn/api/v1/models/codefuse-ai/CodeFuse-QWen-14B/repo?Revision=master&FilePath=LOGO.jpg&View=true" width="800"/>
+<p>
+
+[[中文]](#chinese)    [[English]](#english)
+
+
+
+<a id="english"></a>
+
+## Model Description
+
+CodeFuse-CodeLlama-34B is a 34B Code-LLM finetuned by QLoRA of multiple code tasks（600k instrunctions/answers） on the base model CodeLlama-34b-Python. 
+The context length of finetuning is 4K while it is able to be finetuned by 16k context if necessary.
+<br>
+
+## News and Updates
+
+🔥🔥🔥 CodeFuse-CodeLlama34B-MFT has achived 74.4% of pass@1 on HumanEval, which is SOTA at present.
+
+<br>
+
+## Code Community
+
+**Homepage**: 🏡 https://github.com/codefuse-ai (**Please give us your support with a Star🌟 + Fork🚀 + Watch👀**)
+
++ If you wish to fine-tune the model yourself, you can visit ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨
+
++ If you wish to deploy the model yourself, you can visit ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨
+
++ If you wish to see a demo of the model, you can visit ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨
+
+
+## Performance
+
+| Model                       | HumanEval(pass@1) |  Date   |
+|:----------------------------|:-----------------:|:-------:|
+| **CodeFuse-CodeLlama-34B**  |     **74.4%**      | 2023.9  |
+| WizardCoder-Python-34B-V1.0 |       73.2%       | 2023.8  |
+| GPT-4(zero-shot)            |       67.0%       | 2023.3  |
+| PanGu-Coder2 15B            |       61.6%       | 2023.8  |
+| CodeLlama-34b-Python        |       53.7%       | 2023.8  |
+| CodeLlama-34b               |       48.8%       | 2023.8  |
+| GPT-3.5(zero-shot)          |       48.1%       | 2022.11 |
+| OctoCoder                   |       46.2%       | 2023.8  |
+| StarCoder-15B               |       33.6%       | 2023.5  |
+| LLaMA 2 70B(zero-shot)      |       29.9%       | 2023.7  |
+
+<br>
+
+## Requirements
+
+* python>=3.8 
+* pytorch>=2.0.0
+* transformers==4.32.0
+* Sentencepiece
+* CUDA 11.4
+  <br>
+
+##  Inference String Format
+
+The inference string is a concatenated string formed by combining conversation data(system, human and bot contents) in the training data format.  It is used as input during the inference process.
+Here is an example format of the concatenated string:
+
+```python
+"""
+<|role_start|>system<|role_end|>System instruction
+<|role_start|>human<|role_end|>Human 1st round input
+<|role_start|>bot<|role_end|>Bot 1st round output</s>
+<|role_start|>human<|role_end|>Human 2nd round input
+<|role_start|>bot<|role_end|>Bot 2nd round output</s>
+...
+...
+...
+<|role_start|>human<|role_end|>Human nth round input
+<|role_start|>bot<|role_end|>{Bot output to be genreated}</s>
+"""
+```
+
+When applying inference, you always make your input string end with "<|role_start|>bot<|role_end|>" to ask the model generating answers.
+
+## Quickstart
+
+```bash
+pip install -r requirements.txt
+```
+
+```python
+import torch
+from modelscope import AutoTokenizer, AutoModelForCausalLM, snapshot_download
+
+
+model_dir = snapshot_download('codefuse-ai/CodeFuse-CodeLlama-34B', revision='v1.0.0')
+tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True, use_fast=False, legacy=False)
+tokenizer.padding_side = "left"
+tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("<unk>")
+tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids("</s>")
+model = AutoModelForCausalLM.from_pretrained(model_dir, trust_remote_code=True, 
+                                             device_map='auto', 
+                                             torch_dtype=torch.bfloat16)
+
+HUMAN_ROLE_START_TAG = "<|role_start|>human<|role_end|>"
+BOT_ROLE_START_TAG = "<|role_start|>bot<|role_end|>"
+
+text = f"{HUMAN_ROLE_START_TAG}write a python function of quick sort.{BOT_ROLE_START_TAG}" 
+inputs = tokenizer(text, return_tensors='pt', padding=True, add_special_tokens=False).to("cuda")
+outputs = model.generate(
+        inputs=inputs["input_ids"],
+        attention_mask=inputs["attention_mask"],
+        max_new_tokens=512,
+        top_p=0.95,
+        temperature=0.1,
+        do_sample=True,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id
+    )
+gen_text = tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(gen_text)
+```
+
+## MD5
+We notice that the file may be corrupted during transfer process. Please check MD5 value before use.
+
+| Model File                       | MD5 Value                        |
+|:---------------------------------|:--------------------------------:|
+| pytorch_model-00001-of-00007.bin | 8d544b1bcb3449934184d4141137329c |
+| pytorch_model-00002-of-00007.bin | 9d5dbb30911e48a42fb6d0fcabb322a4 |
+| pytorch_model-00003-of-00007.bin | b0d4aecee0457d9332005a187e1fffed |
+| pytorch_model-00004-of-00007.bin | 5c7e002de5eab77d0194a2b0f6de0c24 |
+| pytorch_model-00005-of-00007.bin | d22a511aa26b5b17117b665a877490ab |
+| pytorch_model-00006-of-00007.bin | a5c28ac277fac07d16dd66537e54d109 |
+| pytorch_model-00007-of-00007.bin | a967e2c6195477b7407089c0bffa2d53 |
+
+
+<a id="chinese"></a>
+
+## 模型简介
+
+CodeFuse-CodeLlama34B-MFT 是一个通过QLoRA对基座模型CodeLlama-34b-Python进行多代码任务微调的代码大模型。模型微调采用了4k上下文。如果有必要，可以扩展到16k。
+<br>
+
+## 新闻
+
+🔥🔥🔥 CodeFuse-CodeLlama34B-MFT模型在HumanEval pass@1上可以达到74.4%, 为当前开源SOTA。
+
+<br>
+
+## 代码社区
+**大本营**： 🏡 https://github.com/codefuse-ai （**欢迎为我们的项目一键三连 Star🌟 + Fork🚀 + Watch👀**）
+
++ 如果您想自己微调该模型，可以访问 ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨
+
++ 如果您想自己部署该模型，可以访问 ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨
+
++ 如果您想观看该模型示例，可以访问 ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨
+
+## 评测表现(代码)
+
+| 模型                          | HumanEval(pass@1) |   日期    |
+|:----------------------------|:-----------------:|:-------:|
+| **CodeFuse-CodeLlama-34B**  |     **74.4%**      | 2023.9  |
+| WizardCoder-Python-34B-V1.0 |       73.2%       | 2023.8  |
+| GPT-4(zero-shot)            |       67.0%       | 2023.3  |
+| PanGu-Coder2 15B            |       61.6%       | 2023.8  |
+| CodeLlama-34b-Python        |       53.7%       | 2023.8  |
+| CodeLlama-34b               |       48.8%       | 2023.8  |
+| GPT-3.5(zero-shot)          |       48.1%       | 2022.11 |
+| OctoCoder                   |       46.2%       | 2023.8  |
+| StarCoder-15B               |       33.6%       | 2023.5  |
+| LLaMA 2 70B(zero-shot)      |       29.9%       | 2023.7  |
+<br>
+
+## Requirements
+
+* python>=3.8 
+* pytorch>=2.0.0
+* transformers==4.32.0
+* CUDA 11.4
+<br>
+
+## 推理数据格式
+
+推理数据为模型在训练数据格式下拼接的字符串形式，它也是推理时输入prompt拼接的方式：
+
+```python
+"""
+<|role_start|>system<|role_end|>这是System指令
+<|role_start|>human<|role_end|>这是第1轮用户输入的问题
+<|role_start|>bot<|role_end|>这是第1轮模型生成的内容</s>
+<|role_start|>human<|role_end|>这是第2轮用户输入的问题
+<|role_start|>bot<|role_end|>这是第2轮模型生成的内容</s>
+...
+...
+...
+<|role_start|>human<|role_end|>这是第n轮用户输入的问题
+<|role_start|>bot<|role_end|>{模型现在要生成的内容}</s>
+"""
+```
+
+推理时，请确保拼接的prompt字符串以"<|role_start|>bot<|role_end|>"结尾，引导模型生成回答。
+
+## 快速使用
+
+```python
+import torch
+from modelscope import AutoTokenizer, AutoModelForCausalLM, snapshot_download
+
+
+model_dir = snapshot_download('codefuse-ai/CodeFuse-CodeLlama-34B', revision='v1.0.0')
+tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True, use_fast=False, legacy=False)
+tokenizer.padding_side = "left"
+tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("<unk>")
+tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids("</s>")
+model = AutoModelForCausalLM.from_pretrained(model_dir, trust_remote_code=True, 
+                                             device_map='auto', 
+                                             torch_dtype=torch.bfloat16)
+
+HUMAN_ROLE_START_TAG = "<|role_start|>human<|role_end|>"
+BOT_ROLE_START_TAG = "<|role_start|>bot<|role_end|>"
+
+text = f"{HUMAN_ROLE_START_TAG}write a python function of quick sort.{BOT_ROLE_START_TAG}" 
+inputs = tokenizer(text, return_tensors='pt', padding=True, add_special_tokens=False).to("cuda")
+outputs = model.generate(
+        inputs=inputs["input_ids"],
+        attention_mask=inputs["attention_mask"],
+        max_new_tokens=512,
+        top_p=0.95,
+        temperature=0.1,
+        do_sample=True,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id
+    )
+gen_text = tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(gen_text)
+```
+
+
+## MD5
+我们发现模型文件可能会在传输过程中损坏，使用前请检查文件MD5值。
+
+| 模型文件                           | MD5值                            |
+|:---------------------------------|:--------------------------------:|
+| pytorch_model-00001-of-00007.bin | 8d544b1bcb3449934184d4141137329c |
+| pytorch_model-00002-of-00007.bin | 9d5dbb30911e48a42fb6d0fcabb322a4 |
+| pytorch_model-00003-of-00007.bin | b0d4aecee0457d9332005a187e1fffed |
+| pytorch_model-00004-of-00007.bin | 5c7e002de5eab77d0194a2b0f6de0c24 |
+| pytorch_model-00005-of-00007.bin | d22a511aa26b5b17117b665a877490ab |
+| pytorch_model-00006-of-00007.bin | a5c28ac277fac07d16dd66537e54d109 |
+| pytorch_model-00007-of-00007.bin | a967e2c6195477b7407089c0bffa2d53 |

config.json

@@ -0,0 +1,33 @@
+{
+  "_name_or_path": "/mnt/user/qumu/download_models/Mixtral-8x7B-v0.1",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token": "</s>",
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mixtral",
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "num_local_experts": 8,
+  "output_router_logits": false,
+  "pad_token": "<unk>",
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000.0,
+  "router_aux_loss_coef": 0.02,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-generation"}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.36.0"
+}

model-00001-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:191b17811aabffe6c8d03838891da1b25bc8d84ddf234b9c9f3ce83b19b7dd1a
+size 4892809584

model-00002-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a4969aac37d4dbeff4b64aac1df6b5e9ee31d06cedfe4d247d378610e82d734
+size 4983004016

model-00003-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:965fa4a588f0fed051c8ac2a9382cb9b7de6efa74c186a89c4205333061f996f
+size 4983004016

model-00004-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a88532ac4017d6b3d5387110e4bb7f336a91d5d3d111ec8d36e3e8e294b8f943
+size 4899035200

model-00005-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c003042591f2066b2dc1a93cd13c82f90d27e2c08d8e295c51506c69e7d8e76
+size 4983004016

model-00006-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:555e0e87593183b4d48e19404e09072463c38b900fcfdfe79f443bd48ad5a6ca
+size 4983004016

model-00007-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ab9412bd17ca9d916ab7e8de924dbe5fc4c8b385479af305388f3bcf70ccc1e
+size 4899035248

model-00008-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9aaa6f5f069e685608d16153633127e9ca7c9f48588c8cbd20c7338bf3e2428f
+size 4983004072

model-00009-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f69ad0e81cdeb18c3ed9335e3ae6c6265097aee1b2baec02e9bcaeef7e23bf5
+size 4983004072

model-00010-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c6121d8590522114c96c74db86c81338849454d6d46fd9f2f54607ba7e1ffab
+size 4899035248

model-00011-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76607abffc26b8b13dfc490b223ca468cd254fc33c37f67df8e7fd5773d008f5
+size 4983004072

model-00012-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fda2ac25fb878944d2fdb3fdd2ca2f9efe62dab2712d00a22b0ebcc0e07c1c0
+size 4983004072

model-00013-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae2f3b21722565542e87634f0c30e76bee7444d666135c56f55943221fe911ca
+size 4983004072

model-00014-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1809ed02ffdaffa332aaae3f594bb7833dea1afbf63075fded5dd51fb7b00445
+size 4899035248

model-00015-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:423811d39b090daa6c3ea1a67c372a48aa5d0595db7838adb2f3aefda3b4a6c6
+size 4983004072

model-00016-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d010892647092933a13306ce1c46c47065621eae6e30f96ee9ba6fec85947d7b
+size 4983004072

model-00017-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2825d1560d444e3970ddd69e743a9ae01b2c07192932e039802c7b761f9f9ebf
+size 4899035248

model-00018-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1fe295fe2dd32f9f4498427df10760073b579d192ee03f1456cc2363323132c1
+size 4983004072

model-00019-of-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a48ac5961e0fe2a3d30d7b570fb267b329095cffe0499b7b3893cf7f480642b6
+size 4221679088

model.safetensors.index.json

@@ -0,0 +1,1002 @@
+{
+  "metadata": {
+    "total_size": 93405585408
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00019-of-00019.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.block_sparse_moe.gate.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00019.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.block_sparse_moe.gate.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00007-of-00019.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.block_sparse_moe.gate.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00008-of-00019.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00007-of-00019.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00007-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.block_sparse_moe.gate.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.block_sparse_moe.gate.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00009-of-00019.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00008-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.block_sparse_moe.gate.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.block_sparse_moe.gate.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00010-of-00019.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00009-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.16.block_sparse_moe.gate.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00011-of-00019.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00011-of-00019.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00010-of-00019.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00010-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.block_sparse_moe.gate.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.block_sparse_moe.gate.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00012-of-00019.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00011-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.block_sparse_moe.gate.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.block_sparse_moe.gate.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.block_sparse_moe.gate.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00013-of-00019.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00012-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.block_sparse_moe.gate.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w2.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.block_sparse_moe.gate.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00014-of-00019.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00013-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w2.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w3.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w1.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.block_sparse_moe.gate.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00015-of-00019.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00014-of-00019.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00014-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.block_sparse_moe.gate.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w2.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.block_sparse_moe.gate.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00016-of-00019.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00015-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.block_sparse_moe.gate.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w3.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w1.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.block_sparse_moe.gate.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00017-of-00019.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00016-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w2.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w3.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w1.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.28.block_sparse_moe.gate.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00018-of-00019.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00018-of-00019.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00017-of-00019.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00017-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.block_sparse_moe.gate.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.block_sparse_moe.gate.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00003-of-00019.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00002-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w1.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w2.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w3.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.block_sparse_moe.gate.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00019-of-00019.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00018-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w1.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w2.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w3.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.block_sparse_moe.gate.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00019-of-00019.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00019-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.block_sparse_moe.gate.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.block_sparse_moe.gate.weight": "model-00003-of-00019.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00004-of-00019.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00003-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.block_sparse_moe.gate.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00005-of-00019.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00004-of-00019.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00004-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.block_sparse_moe.gate.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.block_sparse_moe.gate.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00006-of-00019.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00005-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.block_sparse_moe.gate.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00006-of-00019.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00006-of-00019.safetensors",
+    "model.norm.weight": "model-00019-of-00019.safetensors"
+  }
+}

requirements.txt

@@ -0,0 +1 @@
+git+https://github.com/huggingface/evaluate@8b9373dc8693ffe0244a52551ac5573cffa503aa

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}