Model save

README.md

@@ -0,0 +1,98 @@
+---
+license: mit
+base_model: databricks/dolly-v2-7b
+tags:
+- generated_from_trainer
+model-index:
+- name: dolly-v2-7b-dpo-full-1-epoch-hydrox-safe
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# dolly-v2-7b-dpo-full-1-epoch-hydrox-safe
+
+This model is a fine-tuned version of [databricks/dolly-v2-7b](https://huggingface.co/databricks/dolly-v2-7b) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.0371
+- Rewards/chosen: 4.2799
+- Rewards/rejected: -3.8888
+- Rewards/accuracies: 0.9857
+- Rewards/margins: 8.1686
+- Logps/rejected: -598.4040
+- Logps/chosen: -377.1240
+- Logits/rejected: -1.2002
+- Logits/chosen: -1.5171
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-07
+- train_batch_size: 8
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- total_train_batch_size: 64
+- total_eval_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.618         | 0.03  | 100  | 0.5642          | 0.6988         | -0.1139          | 0.7424             | 0.8127          | -560.6550      | -412.9344    | -1.1894         | -1.4878       |
+| 0.3539        | 0.07  | 200  | 0.3197          | 1.9159         | -0.2730          | 0.8847             | 2.1889          | -562.2463      | -400.7641    | -1.1625         | -1.4800       |
+| 0.2287        | 0.1   | 300  | 0.2128          | 2.8057         | -0.5539          | 0.9200             | 3.3596          | -565.0551      | -391.8654    | -1.1361         | -1.4649       |
+| 0.158         | 0.14  | 400  | 0.1673          | 3.4556         | -1.0339          | 0.9327             | 4.4895          | -569.8558      | -385.3670    | -1.1300         | -1.4622       |
+| 0.1599        | 0.17  | 500  | 0.1397          | 3.7485         | -1.3338          | 0.9461             | 5.0823          | -572.8546      | -382.4376    | -1.1275         | -1.4607       |
+| 0.1389        | 0.2   | 600  | 0.1273          | 3.9259         | -1.5111          | 0.9529             | 5.4371          | -574.6277      | -380.6633    | -1.1194         | -1.4519       |
+| 0.0778        | 0.24  | 700  | 0.1122          | 4.0699         | -1.8498          | 0.9613             | 5.9197          | -578.0140      | -379.2233    | -1.1302         | -1.4542       |
+| 0.0993        | 0.27  | 800  | 0.0975          | 4.2423         | -1.9934          | 0.9663             | 6.2357          | -579.4506      | -377.5001    | -1.1424         | -1.4689       |
+| 0.111         | 0.31  | 900  | 0.0907          | 4.3218         | -2.2534          | 0.9697             | 6.5752          | -582.0501      | -376.7048    | -1.1542         | -1.4820       |
+| 0.0893        | 0.34  | 1000 | 0.0882          | 4.3878         | -2.2588          | 0.9663             | 6.6466          | -582.1047      | -376.0451    | -1.1497         | -1.4694       |
+| 0.079         | 0.37  | 1100 | 0.0840          | 4.4706         | -2.3132          | 0.9689             | 6.7838          | -582.6481      | -375.2164    | -1.1532         | -1.4807       |
+| 0.0706        | 0.41  | 1200 | 0.0721          | 4.4319         | -2.6505          | 0.9722             | 7.0824          | -586.0217      | -375.6038    | -1.1667         | -1.4885       |
+| 0.0705        | 0.44  | 1300 | 0.0725          | 4.3743         | -2.8717          | 0.9739             | 7.2460          | -588.2330      | -376.1799    | -1.1817         | -1.5001       |
+| 0.0537        | 0.48  | 1400 | 0.0648          | 4.3847         | -2.9676          | 0.9756             | 7.3523          | -589.1927      | -376.0760    | -1.1789         | -1.5019       |
+| 0.0483        | 0.51  | 1500 | 0.0604          | 4.3761         | -3.2295          | 0.9798             | 7.6056          | -591.8114      | -376.1613    | -1.1923         | -1.5114       |
+| 0.0572        | 0.54  | 1600 | 0.0581          | 4.3258         | -3.2641          | 0.9773             | 7.5899          | -592.1575      | -376.6645    | -1.1855         | -1.5042       |
+| 0.066         | 0.58  | 1700 | 0.0539          | 4.3270         | -3.3813          | 0.9815             | 7.7083          | -593.3289      | -376.6523    | -1.1886         | -1.5110       |
+| 0.0561        | 0.61  | 1800 | 0.0501          | 4.3859         | -3.3980          | 0.9798             | 7.7839          | -593.4964      | -376.0636    | -1.1948         | -1.5144       |
+| 0.0538        | 0.65  | 1900 | 0.0504          | 4.4209         | -3.4478          | 0.9815             | 7.8687          | -593.9944      | -375.7137    | -1.2036         | -1.5147       |
+| 0.0493        | 0.68  | 2000 | 0.0472          | 4.3835         | -3.5804          | 0.9832             | 7.9639          | -595.3203      | -376.0873    | -1.1925         | -1.5071       |
+| 0.0374        | 0.71  | 2100 | 0.0449          | 4.2972         | -3.7998          | 0.9840             | 8.0970          | -597.5147      | -376.9510    | -1.2020         | -1.5166       |
+| 0.0475        | 0.75  | 2200 | 0.0442          | 4.3073         | -3.6486          | 0.9840             | 7.9559          | -596.0024      | -376.8494    | -1.1992         | -1.5177       |
+| 0.0407        | 0.78  | 2300 | 0.0408          | 4.3011         | -3.7981          | 0.9882             | 8.0992          | -597.4978      | -376.9122    | -1.2078         | -1.5242       |
+| 0.0386        | 0.82  | 2400 | 0.0397          | 4.3423         | -3.7314          | 0.9882             | 8.0737          | -596.8302      | -376.4996    | -1.2029         | -1.5133       |
+| 0.0504        | 0.85  | 2500 | 0.0390          | 4.3732         | -3.7690          | 0.9857             | 8.1422          | -597.2065      | -376.1912    | -1.2024         | -1.5188       |
+| 0.0402        | 0.88  | 2600 | 0.0377          | 4.3358         | -3.8299          | 0.9865             | 8.1656          | -597.8150      | -376.5649    | -1.1977         | -1.5158       |
+| 0.038         | 0.92  | 2700 | 0.0397          | 4.3284         | -3.8383          | 0.9891             | 8.1667          | -597.8990      | -376.6386    | -1.2033         | -1.5139       |
+| 0.0527        | 0.95  | 2800 | 0.0383          | 4.2985         | -3.8490          | 0.9857             | 8.1475          | -598.0059      | -376.9374    | -1.2037         | -1.5196       |
+| 0.0365        | 0.99  | 2900 | 0.0379          | 4.3086         | -3.8349          | 0.9874             | 8.1435          | -597.8653      | -376.8369    | -1.1997         | -1.5156       |
+
+
+### Framework versions
+
+- Transformers 4.35.0
+- Pytorch 2.1.1+cu121
+- Datasets 2.14.6
+- Tokenizers 0.14.1

all_results.json

@@ -0,0 +1,21 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": -1.5171043872833252,
+    "eval_logits/rejected": -1.2001606225967407,
+    "eval_logps/chosen": -377.12396240234375,
+    "eval_logps/rejected": -598.4039916992188,
+    "eval_loss": 0.03705598786473274,
+    "eval_rewards/accuracies": 0.9856902360916138,
+    "eval_rewards/chosen": 4.279876708984375,
+    "eval_rewards/margins": 8.16864013671875,
+    "eval_rewards/rejected": -3.888763189315796,
+    "eval_runtime": 560.6128,
+    "eval_samples": 9500,
+    "eval_samples_per_second": 16.946,
+    "eval_steps_per_second": 0.53,
+    "train_loss": 0.11494619559330763,
+    "train_runtime": 36321.6775,
+    "train_samples": 188284,
+    "train_samples_per_second": 5.184,
+    "train_steps_per_second": 0.081
+}

config.json

@@ -0,0 +1,36 @@
+{
+  "_name_or_path": "databricks/dolly-v2-7b",
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "custom_pipelines": {
+    "text-generation": {
+      "impl": "instruct_pipeline.InstructionTextGenerationPipeline",
+      "pt": "AutoModelForCausalLM",
+      "tf": "TFAutoModelForCausalLM"
+    }
+  },
+  "eos_token_id": 0,
+  "hidden_act": "gelu",
+  "hidden_dropout": 0.0,
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 16384,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neox",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "rope_scaling": null,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 0.25,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.35.0",
+  "use_cache": false,
+  "use_parallel_residual": true,
+  "vocab_size": 50280
+}

eval_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": -1.5171043872833252,
+    "eval_logits/rejected": -1.2001606225967407,
+    "eval_logps/chosen": -377.12396240234375,
+    "eval_logps/rejected": -598.4039916992188,
+    "eval_loss": 0.03705598786473274,
+    "eval_rewards/accuracies": 0.9856902360916138,
+    "eval_rewards/chosen": 4.279876708984375,
+    "eval_rewards/margins": 8.16864013671875,
+    "eval_rewards/rejected": -3.888763189315796,
+    "eval_runtime": 560.6128,
+    "eval_samples": 9500,
+    "eval_samples_per_second": 16.946,
+    "eval_steps_per_second": 0.53
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.35.0",
+  "use_cache": false
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c1d06cb95d6e9c6612f79d001dcae91a529ceffad6db0f75a73e4a1506a9617
+size 4976550272

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:998cfec6a081877415fc55210949bbba45af5a51634d8d6648733ccc55ccf5bd
+size 4967384136

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78f7f9ed62c7caea6e1f6117eb57c702443cea34bccd701b3449b7ee9eadbdab
+size 3768225552

model.safetensors.index.json

@@ -0,0 +1,395 @@
+{
+  "metadata": {
+    "total_size": 13712113664
+  },
+  "weight_map": {
+    "embed_out.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.embed_in.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.final_layer_norm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.final_layer_norm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.0.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.10.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.11.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.11.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.11.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.11.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.11.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.12.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.2.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.20.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.mlp.dense_4h_to_h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.mlp.dense_4h_to_h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.22.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.attention.dense.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.attention.dense.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.attention.query_key_value.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.attention.query_key_value.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.input_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.23.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.23.mlp.dense_h_to_4h.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.mlp.dense_h_to_4h.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.post_attention_layernorm.bias": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.23.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "gpt_neox.layers.24.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.3.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.30.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.attention.dense.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.attention.dense.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.attention.query_key_value.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.attention.query_key_value.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.input_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.mlp.dense_4h_to_h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.mlp.dense_4h_to_h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.mlp.dense_h_to_4h.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.mlp.dense_h_to_4h.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.post_attention_layernorm.bias": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "gpt_neox.layers.4.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.attention.dense.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.attention.dense.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.attention.query_key_value.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.attention.query_key_value.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.input_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.mlp.dense_4h_to_h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.mlp.dense_4h_to_h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.mlp.dense_h_to_4h.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.mlp.dense_h_to_4h.weight": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.post_attention_layernorm.bias": "model-00001-of-00003.safetensors",
+    "gpt_neox.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors"
+  }
+}

runs/Jan16_22-47-28_amaterasu/events.out.tfevents.1705445675.amaterasu.3245306.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4b6a0b117a164c6fd725a0959c9fdf3b4adf0d479ae072cda46729a115a3eb0
+size 213354

runs/Jan16_22-47-28_amaterasu/events.out.tfevents.1705482557.amaterasu.3245306.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22510410c0a9bf7d33044f2879efbc2691489987294a65e0b29fc419d3038904
+size 828

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "additional_special_tokens": [
+    "### End",
+    "### Instruction:",
+    "### Response:"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,242 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "### End",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50278": {
+      "content": "### Instruction:",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50279": {
+      "content": "### Response:",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "### End",
+    "### Instruction:",
+    "### Response:"
+  ],
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "train_loss": 0.11494619559330763,
+    "train_runtime": 36321.6775,
+    "train_samples": 188284,
+    "train_samples_per_second": 5.184,
+    "train_steps_per_second": 0.081
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:194823f1673c19dbfdbf3a6a2e5f4d6442f7943f325a75441b204de220d1533d
+size 5752