Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,8 +1,202 @@
 ---
-datasets:
-- BounharAbdelaziz/AL-Atlas-Moroccan-Darija-Pretraining-Dataset
-language:
-- ar
-base_model:
-- google/gemma-2-2b
----
+base_model: google/gemma-2-2b
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/gemma-2-2b",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d0e0707e43a9e9fe7d2f680b16f042b41e8e8a07c4c7dce34dcf3f2c2d39b95
+size 51132072

config.json

@@ -0,0 +1,33 @@
+{
+  "_name_or_path": "google/gemma-2-2b",
+  "architectures": [
+    "Gemma2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": 50.0,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": 1,
+  "final_logit_softcapping": 30.0,
+  "head_dim": 256,
+  "hidden_act": "gelu_pytorch_tanh",
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 2304,
+  "initializer_range": 0.02,
+  "intermediate_size": 9216,
+  "max_position_embeddings": 8192,
+  "model_type": "gemma2",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 26,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "torch_dtype": "float16",
+  "transformers_version": "4.46.3",
+  "use_cache": true,
+  "vocab_size": 256000
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dda9cf296b67e694f8058fbc420ad9891df7d5fe78a2bf4cf4791dddc2a8ca78
+size 4988025488

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07800b1e30a60616187dd6ffdc7ffaeaa002b792d17dee431c08017b67880c39
+size 240691712

model.safetensors.index.json

@@ -0,0 +1,295 @@
+{
+  "metadata": {
+    "total_size": 5228683776
+  },
+  "weight_map": {
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.pre_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.pre_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4fddafdc0a262675c005f0fb1a82f0dc59ef45e773b2b4554bbdbe475b4f8c3
+size 102300346

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d81f0f10258c1cba56da4c28c08d82046c7a091d96fe74cf2719a38ac82e7cfe
+size 14244

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c8bc5ca62c62ea2472d0194e675c1232665d8163971e96e1dc15b94be793fc1
+size 1064

special_tokens_map.json

@@ -0,0 +1,28 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<eos>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f7eee611703c5ce5d1eee32d9cdcfe465647b8aff0c1dfb3bed7ad7dbb05060
+size 34362873

tokenizer_config.json

@@ -0,0 +1,2012 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "5": {
+      "content": "<2mass>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "6": {
+      "content": "[@BOS@]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "7": {
+      "content": "<unused0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "<unused1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "<unused2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "<unused3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "<unused4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "<unused5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "<unused6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "<unused7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "15": {
+      "content": "<unused8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "16": {
+      "content": "<unused9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "17": {
+      "content": "<unused10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "18": {
+      "content": "<unused11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "19": {
+      "content": "<unused12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "20": {
+      "content": "<unused13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "21": {
+      "content": "<unused14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "22": {
+      "content": "<unused15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "23": {
+      "content": "<unused16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "24": {
+      "content": "<unused17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "25": {
+      "content": "<unused18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "26": {
+      "content": "<unused19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "27": {
+      "content": "<unused20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "28": {
+      "content": "<unused21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "29": {
+      "content": "<unused22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "30": {
+      "content": "<unused23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "31": {
+      "content": "<unused24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32": {
+      "content": "<unused25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "33": {
+      "content": "<unused26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "34": {
+      "content": "<unused27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "35": {
+      "content": "<unused28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "36": {
+      "content": "<unused29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "37": {
+      "content": "<unused30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "38": {
+      "content": "<unused31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "39": {
+      "content": "<unused32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "40": {
+      "content": "<unused33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "41": {
+      "content": "<unused34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "42": {
+      "content": "<unused35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "43": {
+      "content": "<unused36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "44": {
+      "content": "<unused37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "45": {
+      "content": "<unused38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "46": {
+      "content": "<unused39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "47": {
+      "content": "<unused40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "48": {
+      "content": "<unused41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49": {
+      "content": "<unused42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50": {
+      "content": "<unused43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "51": {
+      "content": "<unused44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "52": {
+      "content": "<unused45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "53": {
+      "content": "<unused46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "54": {
+      "content": "<unused47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "55": {
+      "content": "<unused48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "56": {
+      "content": "<unused49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "57": {
+      "content": "<unused50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "58": {
+      "content": "<unused51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "59": {
+      "content": "<unused52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "60": {
+      "content": "<unused53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "61": {
+      "content": "<unused54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "62": {
+      "content": "<unused55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "63": {
+      "content": "<unused56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "64": {
+      "content": "<unused57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65": {
+      "content": "<unused58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "66": {
+      "content": "<unused59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "67": {
+      "content": "<unused60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "68": {
+      "content": "<unused61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "69": {
+      "content": "<unused62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "70": {
+      "content": "<unused63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "71": {
+      "content": "<unused64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "72": {
+      "content": "<unused65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "73": {
+      "content": "<unused66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "74": {
+      "content": "<unused67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "75": {
+      "content": "<unused68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "76": {
+      "content": "<unused69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "77": {
+      "content": "<unused70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "78": {
+      "content": "<unused71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79": {
+      "content": "<unused72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "80": {
+      "content": "<unused73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "81": {
+      "content": "<unused74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "82": {
+      "content": "<unused75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "83": {
+      "content": "<unused76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "84": {
+      "content": "<unused77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "85": {
+      "content": "<unused78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "86": {
+      "content": "<unused79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "87": {
+      "content": "<unused80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "88": {
+      "content": "<unused81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "89": {
+      "content": "<unused82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "90": {
+      "content": "<unused83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "91": {
+      "content": "<unused84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "92": {
+      "content": "<unused85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "93": {
+      "content": "<unused86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "94": {
+      "content": "<unused87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "95": {
+      "content": "<unused88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "96": {
+      "content": "<unused89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "97": {
+      "content": "<unused90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "98": {
+      "content": "<unused91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "99": {
+      "content": "<unused92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100": {
+      "content": "<unused93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "101": {
+      "content": "<unused94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102": {
+      "content": "<unused95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "103": {
+      "content": "<unused96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "104": {
+      "content": "<unused97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "105": {
+      "content": "<unused98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "106": {
+      "content": "<start_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<end_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "109": {
+      "content": "\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "110": {
+      "content": "\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "111": {
+      "content": "\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "112": {
+      "content": "\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "113": {
+      "content": "\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "114": {
+      "content": "\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "115": {
+      "content": "\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "116": {
+      "content": "\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "117": {
+      "content": "\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "118": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "119": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "120": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "121": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "122": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "123": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "124": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "125": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "126": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "127": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "129": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "130": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "131": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "132": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "133": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "134": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "135": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "136": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "137": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "138": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "139": {
+      "content": "▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "140": {
+      "content": "▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "141": {
+      "content": "▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "142": {
+      "content": "▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "143": {
+      "content": "▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "144": {
+      "content": "▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "145": {
+      "content": "▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "146": {
+      "content": "▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "147": {
+      "content": "▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "148": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "149": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "150": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "152": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "153": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "154": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "155": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "156": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "157": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "158": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "159": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "160": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "161": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "162": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "163": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "164": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "165": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "166": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "167": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "168": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "169": {
+      "content": "<table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "170": {
+      "content": "<caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "171": {
+      "content": "<thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "172": {
+      "content": "<tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "173": {
+      "content": "<tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "174": {
+      "content": "<tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "175": {
+      "content": "<th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "176": {
+      "content": "<td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "177": {
+      "content": "</table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "178": {
+      "content": "</caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "179": {
+      "content": "</thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "180": {
+      "content": "</tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "181": {
+      "content": "</tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "182": {
+      "content": "</tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "183": {
+      "content": "</th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "184": {
+      "content": "</td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "185": {
+      "content": "<h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "186": {
+      "content": "<h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "187": {
+      "content": "<h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "188": {
+      "content": "<h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "189": {
+      "content": "<h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "190": {
+      "content": "<h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "191": {
+      "content": "<blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "192": {
+      "content": "</h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "193": {
+      "content": "</h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "194": {
+      "content": "</h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "195": {
+      "content": "</h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "196": {
+      "content": "</h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "197": {
+      "content": "</h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "198": {
+      "content": "</blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "199": {
+      "content": "<strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200": {
+      "content": "<em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "201": {
+      "content": "<b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "202": {
+      "content": "<i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "203": {
+      "content": "<u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "204": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "205": {
+      "content": "<sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "206": {
+      "content": "<sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "207": {
+      "content": "<code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "208": {
+      "content": "</strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "209": {
+      "content": "</em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "210": {
+      "content": "</b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "211": {
+      "content": "</i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "212": {
+      "content": "</u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "213": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "214": {
+      "content": "</sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "215": {
+      "content": "</sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "216": {
+      "content": "</code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255968": {
+      "content": "[toxicity=0]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255969": {
+      "content": "\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255970": {
+      "content": "\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255971": {
+      "content": "\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255972": {
+      "content": "\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255973": {
+      "content": "\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255974": {
+      "content": "\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255975": {
+      "content": "\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255976": {
+      "content": "\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255977": {
+      "content": "\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255978": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255979": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255980": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255981": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255982": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255983": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255984": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255985": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255986": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255987": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255988": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255989": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255990": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255991": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255992": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255993": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255994": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255995": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255996": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255997": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255998": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255999": {
+      "content": "<unused99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<eos>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

trainer_state.json

@@ -0,0 +1,2427 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 34247,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0029199637924489734,
+      "grad_norm": NaN,
+      "learning_rate": 1.906614785992218e-05,
+      "loss": 4.8545,
+      "step": 100
+    },
+    {
+      "epoch": 0.005839927584897947,
+      "grad_norm": 2.070359230041504,
+      "learning_rate": 3.852140077821012e-05,
+      "loss": 4.5735,
+      "step": 200
+    },
+    {
+      "epoch": 0.008759891377346922,
+      "grad_norm": 2.1655333042144775,
+      "learning_rate": 5.7976653696498064e-05,
+      "loss": 4.2565,
+      "step": 300
+    },
+    {
+      "epoch": 0.011679855169795894,
+      "grad_norm": 3.0300276279449463,
+      "learning_rate": 7.7431906614786e-05,
+      "loss": 4.157,
+      "step": 400
+    },
+    {
+      "epoch": 0.014599818962244868,
+      "grad_norm": 2.706392526626587,
+      "learning_rate": 9.688715953307394e-05,
+      "loss": 4.0463,
+      "step": 500
+    },
+    {
+      "epoch": 0.017519782754693843,
+      "grad_norm": 3.1594862937927246,
+      "learning_rate": 0.00011634241245136189,
+      "loss": 3.9791,
+      "step": 600
+    },
+    {
+      "epoch": 0.020439746547142815,
+      "grad_norm": 3.447601556777954,
+      "learning_rate": 0.0001357976653696498,
+      "loss": 3.9192,
+      "step": 700
+    },
+    {
+      "epoch": 0.023359710339591788,
+      "grad_norm": 2.8876287937164307,
+      "learning_rate": 0.00015525291828793775,
+      "loss": 3.9025,
+      "step": 800
+    },
+    {
+      "epoch": 0.026279674132040763,
+      "grad_norm": 2.7174556255340576,
+      "learning_rate": 0.0001747081712062257,
+      "loss": 3.7831,
+      "step": 900
+    },
+    {
+      "epoch": 0.029199637924489735,
+      "grad_norm": 3.173799753189087,
+      "learning_rate": 0.00019416342412451363,
+      "loss": 3.7109,
+      "step": 1000
+    },
+    {
+      "epoch": 0.03211960171693871,
+      "grad_norm": 3.4008660316467285,
+      "learning_rate": 0.00019999780875400972,
+      "loss": 3.6651,
+      "step": 1100
+    },
+    {
+      "epoch": 0.035039565509387686,
+      "grad_norm": 3.263437509536743,
+      "learning_rate": 0.00019998707635176594,
+      "loss": 3.5635,
+      "step": 1200
+    },
+    {
+      "epoch": 0.03795952930183666,
+      "grad_norm": 4.769214153289795,
+      "learning_rate": 0.000199967401278203,
+      "loss": 3.4451,
+      "step": 1300
+    },
+    {
+      "epoch": 0.04087949309428563,
+      "grad_norm": 3.8638391494750977,
+      "learning_rate": 0.0001999387852930255,
+      "loss": 3.406,
+      "step": 1400
+    },
+    {
+      "epoch": 0.0437994568867346,
+      "grad_norm": 3.9781579971313477,
+      "learning_rate": 0.00019990123095559763,
+      "loss": 3.3305,
+      "step": 1500
+    },
+    {
+      "epoch": 0.046719420679183575,
+      "grad_norm": 4.234069347381592,
+      "learning_rate": 0.00019985474162471448,
+      "loss": 3.2528,
+      "step": 1600
+    },
+    {
+      "epoch": 0.049639384471632554,
+      "grad_norm": 3.5920116901397705,
+      "learning_rate": 0.00019979932145830148,
+      "loss": 3.1945,
+      "step": 1700
+    },
+    {
+      "epoch": 0.052559348264081526,
+      "grad_norm": 4.009890556335449,
+      "learning_rate": 0.00019973497541304251,
+      "loss": 3.0998,
+      "step": 1800
+    },
+    {
+      "epoch": 0.0554793120565305,
+      "grad_norm": 3.665900707244873,
+      "learning_rate": 0.00019966170924393665,
+      "loss": 2.9929,
+      "step": 1900
+    },
+    {
+      "epoch": 0.05839927584897947,
+      "grad_norm": 5.364969730377197,
+      "learning_rate": 0.00019957952950378348,
+      "loss": 2.9956,
+      "step": 2000
+    },
+    {
+      "epoch": 0.06131923964142845,
+      "grad_norm": 5.028293132781982,
+      "learning_rate": 0.00019948844354259694,
+      "loss": 2.9441,
+      "step": 2100
+    },
+    {
+      "epoch": 0.06423920343387741,
+      "grad_norm": 4.162110328674316,
+      "learning_rate": 0.0001993895033640701,
+      "loss": 2.8629,
+      "step": 2200
+    },
+    {
+      "epoch": 0.0671591672263264,
+      "grad_norm": 5.760902404785156,
+      "learning_rate": 0.0001992807190408027,
+      "loss": 2.8007,
+      "step": 2300
+    },
+    {
+      "epoch": 0.07007913101877537,
+      "grad_norm": 5.505645751953125,
+      "learning_rate": 0.00019916305522159333,
+      "loss": 2.7621,
+      "step": 2400
+    },
+    {
+      "epoch": 0.07299909481122434,
+      "grad_norm": 4.4763946533203125,
+      "learning_rate": 0.00019903652243009065,
+      "loss": 2.7013,
+      "step": 2500
+    },
+    {
+      "epoch": 0.07591905860367332,
+      "grad_norm": 4.271712779998779,
+      "learning_rate": 0.00019890113198316889,
+      "loss": 2.618,
+      "step": 2600
+    },
+    {
+      "epoch": 0.07883902239612228,
+      "grad_norm": 5.071974277496338,
+      "learning_rate": 0.00019875689598991564,
+      "loss": 2.5495,
+      "step": 2700
+    },
+    {
+      "epoch": 0.08175898618857126,
+      "grad_norm": 4.088096618652344,
+      "learning_rate": 0.0001986038273505489,
+      "loss": 2.4626,
+      "step": 2800
+    },
+    {
+      "epoch": 0.08467894998102024,
+      "grad_norm": 6.447364807128906,
+      "learning_rate": 0.0001984419397552632,
+      "loss": 2.455,
+      "step": 2900
+    },
+    {
+      "epoch": 0.0875989137734692,
+      "grad_norm": 5.120512008666992,
+      "learning_rate": 0.0001982712476830054,
+      "loss": 2.4463,
+      "step": 3000
+    },
+    {
+      "epoch": 0.09051887756591818,
+      "grad_norm": 4.25977087020874,
+      "learning_rate": 0.0001980917664001794,
+      "loss": 2.3021,
+      "step": 3100
+    },
+    {
+      "epoch": 0.09343884135836715,
+      "grad_norm": 7.021845817565918,
+      "learning_rate": 0.0001979035119592809,
+      "loss": 2.2682,
+      "step": 3200
+    },
+    {
+      "epoch": 0.09635880515081613,
+      "grad_norm": 5.458634853363037,
+      "learning_rate": 0.00019770650119746192,
+      "loss": 2.2192,
+      "step": 3300
+    },
+    {
+      "epoch": 0.09927876894326511,
+      "grad_norm": 7.42665433883667,
+      "learning_rate": 0.00019750075173502444,
+      "loss": 2.1495,
+      "step": 3400
+    },
+    {
+      "epoch": 0.10219873273571407,
+      "grad_norm": 5.143280029296875,
+      "learning_rate": 0.00019728628197384485,
+      "loss": 2.1331,
+      "step": 3500
+    },
+    {
+      "epoch": 0.10511869652816305,
+      "grad_norm": 6.114063739776611,
+      "learning_rate": 0.00019706311109572792,
+      "loss": 2.1047,
+      "step": 3600
+    },
+    {
+      "epoch": 0.10803866032061202,
+      "grad_norm": 7.758254051208496,
+      "learning_rate": 0.00019683362048532321,
+      "loss": 2.0496,
+      "step": 3700
+    },
+    {
+      "epoch": 0.110958624113061,
+      "grad_norm": 5.546494483947754,
+      "learning_rate": 0.00019659319452882684,
+      "loss": 1.9042,
+      "step": 3800
+    },
+    {
+      "epoch": 0.11387858790550998,
+      "grad_norm": 6.260044574737549,
+      "learning_rate": 0.0001963441294439365,
+      "loss": 1.9357,
+      "step": 3900
+    },
+    {
+      "epoch": 0.11679855169795894,
+      "grad_norm": 6.014352321624756,
+      "learning_rate": 0.00019608644750660297,
+      "loss": 1.8152,
+      "step": 4000
+    },
+    {
+      "epoch": 0.11971851549040792,
+      "grad_norm": 4.853186130523682,
+      "learning_rate": 0.00019582017176345346,
+      "loss": 1.7449,
+      "step": 4100
+    },
+    {
+      "epoch": 0.1226384792828569,
+      "grad_norm": 4.977631568908691,
+      "learning_rate": 0.0001955453260297302,
+      "loss": 1.6956,
+      "step": 4200
+    },
+    {
+      "epoch": 0.12555844307530586,
+      "grad_norm": 3.9587907791137695,
+      "learning_rate": 0.0001952619348871607,
+      "loss": 1.6859,
+      "step": 4300
+    },
+    {
+      "epoch": 0.12847840686775483,
+      "grad_norm": 6.316972255706787,
+      "learning_rate": 0.000194970023681759,
+      "loss": 1.6503,
+      "step": 4400
+    },
+    {
+      "epoch": 0.13139837066020382,
+      "grad_norm": 5.7269368171691895,
+      "learning_rate": 0.0001946726645306457,
+      "loss": 1.6513,
+      "step": 4500
+    },
+    {
+      "epoch": 0.1343183344526528,
+      "grad_norm": 6.6521525382995605,
+      "learning_rate": 0.00019436387681875253,
+      "loss": 1.6014,
+      "step": 4600
+    },
+    {
+      "epoch": 0.13723829824510175,
+      "grad_norm": 7.724483013153076,
+      "learning_rate": 0.0001940466493647889,
+      "loss": 1.4944,
+      "step": 4700
+    },
+    {
+      "epoch": 0.14015826203755075,
+      "grad_norm": 7.2344746589660645,
+      "learning_rate": 0.00019372101054103004,
+      "loss": 1.4505,
+      "step": 4800
+    },
+    {
+      "epoch": 0.1430782258299997,
+      "grad_norm": 6.845386981964111,
+      "learning_rate": 0.0001933869894720496,
+      "loss": 1.4348,
+      "step": 4900
+    },
+    {
+      "epoch": 0.14599818962244868,
+      "grad_norm": 6.9420552253723145,
+      "learning_rate": 0.0001930446160321148,
+      "loss": 1.389,
+      "step": 5000
+    },
+    {
+      "epoch": 0.14891815341489764,
+      "grad_norm": 4.138674736022949,
+      "learning_rate": 0.00019269392084251438,
+      "loss": 1.3654,
+      "step": 5100
+    },
+    {
+      "epoch": 0.15183811720734663,
+      "grad_norm": 3.841850519180298,
+      "learning_rate": 0.00019233493526882008,
+      "loss": 1.3368,
+      "step": 5200
+    },
+    {
+      "epoch": 0.1547580809997956,
+      "grad_norm": 4.870452880859375,
+      "learning_rate": 0.00019196769141808124,
+      "loss": 1.3552,
+      "step": 5300
+    },
+    {
+      "epoch": 0.15767804479224456,
+      "grad_norm": 5.648431301116943,
+      "learning_rate": 0.00019159222213595325,
+      "loss": 1.3289,
+      "step": 5400
+    },
+    {
+      "epoch": 0.16059800858469356,
+      "grad_norm": 6.571835041046143,
+      "learning_rate": 0.0001912085610037598,
+      "loss": 1.2027,
+      "step": 5500
+    },
+    {
+      "epoch": 0.16351797237714252,
+      "grad_norm": 4.793326377868652,
+      "learning_rate": 0.00019081674233548964,
+      "loss": 1.2058,
+      "step": 5600
+    },
+    {
+      "epoch": 0.1664379361695915,
+      "grad_norm": 6.252863883972168,
+      "learning_rate": 0.00019041680117472726,
+      "loss": 1.1419,
+      "step": 5700
+    },
+    {
+      "epoch": 0.16935789996204048,
+      "grad_norm": 5.3782124519348145,
+      "learning_rate": 0.00019000877329151893,
+      "loss": 1.1386,
+      "step": 5800
+    },
+    {
+      "epoch": 0.17227786375448945,
+      "grad_norm": 8.507604598999023,
+      "learning_rate": 0.0001895926951791734,
+      "loss": 1.1592,
+      "step": 5900
+    },
+    {
+      "epoch": 0.1751978275469384,
+      "grad_norm": 7.945569038391113,
+      "learning_rate": 0.00018916860405099808,
+      "loss": 1.0545,
+      "step": 6000
+    },
+    {
+      "epoch": 0.1781177913393874,
+      "grad_norm": 6.228450775146484,
+      "learning_rate": 0.00018873653783697055,
+      "loss": 1.1276,
+      "step": 6100
+    },
+    {
+      "epoch": 0.18103775513183637,
+      "grad_norm": 4.703051567077637,
+      "learning_rate": 0.0001882965351803463,
+      "loss": 1.0775,
+      "step": 6200
+    },
+    {
+      "epoch": 0.18395771892428534,
+      "grad_norm": 7.743706226348877,
+      "learning_rate": 0.00018784863543420258,
+      "loss": 1.0611,
+      "step": 6300
+    },
+    {
+      "epoch": 0.1868776827167343,
+      "grad_norm": 7.018481254577637,
+      "learning_rate": 0.0001873928786579187,
+      "loss": 0.9811,
+      "step": 6400
+    },
+    {
+      "epoch": 0.1897976465091833,
+      "grad_norm": 6.171621799468994,
+      "learning_rate": 0.00018692930561359307,
+      "loss": 1.0307,
+      "step": 6500
+    },
+    {
+      "epoch": 0.19271761030163226,
+      "grad_norm": 9.385793685913086,
+      "learning_rate": 0.0001864579577623977,
+      "loss": 1.0374,
+      "step": 6600
+    },
+    {
+      "epoch": 0.19563757409408122,
+      "grad_norm": 10.805081367492676,
+      "learning_rate": 0.00018597887726086985,
+      "loss": 1.0033,
+      "step": 6700
+    },
+    {
+      "epoch": 0.19855753788653022,
+      "grad_norm": 4.207973957061768,
+      "learning_rate": 0.00018549210695714162,
+      "loss": 0.9768,
+      "step": 6800
+    },
+    {
+      "epoch": 0.20147750167897918,
+      "grad_norm": 4.130248069763184,
+      "learning_rate": 0.0001849976903871079,
+      "loss": 0.9005,
+      "step": 6900
+    },
+    {
+      "epoch": 0.20439746547142815,
+      "grad_norm": 4.633413791656494,
+      "learning_rate": 0.00018449567177053215,
+      "loss": 0.9012,
+      "step": 7000
+    },
+    {
+      "epoch": 0.20731742926387714,
+      "grad_norm": 4.062277793884277,
+      "learning_rate": 0.000183986096007092,
+      "loss": 0.8939,
+      "step": 7100
+    },
+    {
+      "epoch": 0.2102373930563261,
+      "grad_norm": 8.527688026428223,
+      "learning_rate": 0.00018346900867236305,
+      "loss": 0.9296,
+      "step": 7200
+    },
+    {
+      "epoch": 0.21315735684877507,
+      "grad_norm": 8.69731616973877,
+      "learning_rate": 0.000182944456013743,
+      "loss": 0.885,
+      "step": 7300
+    },
+    {
+      "epoch": 0.21607732064122404,
+      "grad_norm": 5.987290382385254,
+      "learning_rate": 0.00018241248494631505,
+      "loss": 0.8927,
+      "step": 7400
+    },
+    {
+      "epoch": 0.21899728443367303,
+      "grad_norm": 4.5624566078186035,
+      "learning_rate": 0.00018187314304865225,
+      "loss": 0.7979,
+      "step": 7500
+    },
+    {
+      "epoch": 0.221917248226122,
+      "grad_norm": 6.830573081970215,
+      "learning_rate": 0.00018132647855856188,
+      "loss": 0.7968,
+      "step": 7600
+    },
+    {
+      "epoch": 0.22483721201857096,
+      "grad_norm": 6.950575828552246,
+      "learning_rate": 0.00018077254036877118,
+      "loss": 0.8561,
+      "step": 7700
+    },
+    {
+      "epoch": 0.22775717581101995,
+      "grad_norm": 3.847403049468994,
+      "learning_rate": 0.00018021137802255458,
+      "loss": 0.773,
+      "step": 7800
+    },
+    {
+      "epoch": 0.23067713960346892,
+      "grad_norm": 4.9719133377075195,
+      "learning_rate": 0.0001796430417093026,
+      "loss": 0.7673,
+      "step": 7900
+    },
+    {
+      "epoch": 0.23359710339591788,
+      "grad_norm": 6.954914569854736,
+      "learning_rate": 0.0001790675822600329,
+      "loss": 0.8196,
+      "step": 8000
+    },
+    {
+      "epoch": 0.23651706718836688,
+      "grad_norm": 5.7400336265563965,
+      "learning_rate": 0.00017848505114284423,
+      "loss": 0.7128,
+      "step": 8100
+    },
+    {
+      "epoch": 0.23943703098081584,
+      "grad_norm": 6.180181503295898,
+      "learning_rate": 0.00017789550045831294,
+      "loss": 0.8407,
+      "step": 8200
+    },
+    {
+      "epoch": 0.2423569947732648,
+      "grad_norm": 7.137686729431152,
+      "learning_rate": 0.00017729898293483348,
+      "loss": 0.7785,
+      "step": 8300
+    },
+    {
+      "epoch": 0.2452769585657138,
+      "grad_norm": 3.9680042266845703,
+      "learning_rate": 0.00017669555192390235,
+      "loss": 0.7693,
+      "step": 8400
+    },
+    {
+      "epoch": 0.24819692235816276,
+      "grad_norm": 6.744853496551514,
+      "learning_rate": 0.00017608526139534626,
+      "loss": 0.7528,
+      "step": 8500
+    },
+    {
+      "epoch": 0.25111688615061173,
+      "grad_norm": 3.3069093227386475,
+      "learning_rate": 0.00017546816593249552,
+      "loss": 0.7815,
+      "step": 8600
+    },
+    {
+      "epoch": 0.2540368499430607,
+      "grad_norm": 4.526154041290283,
+      "learning_rate": 0.00017484432072730185,
+      "loss": 0.7127,
+      "step": 8700
+    },
+    {
+      "epoch": 0.25695681373550966,
+      "grad_norm": NaN,
+      "learning_rate": 0.00017422011991728064,
+      "loss": 0.7122,
+      "step": 8800
+    },
+    {
+      "epoch": 0.2598767775279587,
+      "grad_norm": 4.636015892028809,
+      "learning_rate": 0.00017358300930742573,
+      "loss": 0.7137,
+      "step": 8900
+    },
+    {
+      "epoch": 0.26279674132040765,
+      "grad_norm": 5.9968180656433105,
+      "learning_rate": 0.0001729393175603787,
+      "loss": 0.7437,
+      "step": 9000
+    },
+    {
+      "epoch": 0.2657167051128566,
+      "grad_norm": 3.8191442489624023,
+      "learning_rate": 0.00017228910224681697,
+      "loss": 0.7169,
+      "step": 9100
+    },
+    {
+      "epoch": 0.2686366689053056,
+      "grad_norm": 7.120567321777344,
+      "learning_rate": 0.00017163242152087461,
+      "loss": 0.7889,
+      "step": 9200
+    },
+    {
+      "epoch": 0.27155663269775454,
+      "grad_norm": 6.114504337310791,
+      "learning_rate": 0.0001709693341149409,
+      "loss": 0.7305,
+      "step": 9300
+    },
+    {
+      "epoch": 0.2744765964902035,
+      "grad_norm": 5.748324394226074,
+      "learning_rate": 0.00017029989933440754,
+      "loss": 0.6722,
+      "step": 9400
+    },
+    {
+      "epoch": 0.27739656028265247,
+      "grad_norm": 9.556855201721191,
+      "learning_rate": 0.00016962417705236436,
+      "loss": 0.7727,
+      "step": 9500
+    },
+    {
+      "epoch": 0.2803165240751015,
+      "grad_norm": 6.288482189178467,
+      "learning_rate": 0.00016894907782189354,
+      "loss": 0.725,
+      "step": 9600
+    },
+    {
+      "epoch": 0.28323648786755046,
+      "grad_norm": 8.286351203918457,
+      "learning_rate": 0.00016826102375707788,
+      "loss": 0.7074,
+      "step": 9700
+    },
+    {
+      "epoch": 0.2861564516599994,
+      "grad_norm": 6.121710777282715,
+      "learning_rate": 0.00016756686454426069,
+      "loss": 0.7093,
+      "step": 9800
+    },
+    {
+      "epoch": 0.2890764154524484,
+      "grad_norm": 4.868896007537842,
+      "learning_rate": 0.00016686666226784234,
+      "loss": 0.6829,
+      "step": 9900
+    },
+    {
+      "epoch": 0.29199637924489735,
+      "grad_norm": 3.8444511890411377,
+      "learning_rate": 0.00016616047955270445,
+      "loss": 0.6467,
+      "step": 10000
+    },
+    {
+      "epoch": 0.2949163430373463,
+      "grad_norm": 6.739651679992676,
+      "learning_rate": 0.00016544837955860867,
+      "loss": 0.6722,
+      "step": 10100
+    },
+    {
+      "epoch": 0.2978363068297953,
+      "grad_norm": 6.124034404754639,
+      "learning_rate": 0.00016473042597454782,
+      "loss": 0.7016,
+      "step": 10200
+    },
+    {
+      "epoch": 0.3007562706222443,
+      "grad_norm": 7.008491039276123,
+      "learning_rate": 0.00016400668301304983,
+      "loss": 0.6901,
+      "step": 10300
+    },
+    {
+      "epoch": 0.30367623441469327,
+      "grad_norm": 3.4143664836883545,
+      "learning_rate": 0.00016327721540443436,
+      "loss": 0.6092,
+      "step": 10400
+    },
+    {
+      "epoch": 0.30659619820714223,
+      "grad_norm": 2.6440017223358154,
+      "learning_rate": 0.00016254208839102386,
+      "loss": 0.6216,
+      "step": 10500
+    },
+    {
+      "epoch": 0.3095161619995912,
+      "grad_norm": 6.059576511383057,
+      "learning_rate": 0.0001618013677213079,
+      "loss": 0.6949,
+      "step": 10600
+    },
+    {
+      "epoch": 0.31243612579204016,
+      "grad_norm": 6.078047752380371,
+      "learning_rate": 0.0001610551196440632,
+      "loss": 0.6841,
+      "step": 10700
+    },
+    {
+      "epoch": 0.31535608958448913,
+      "grad_norm": 5.9573655128479,
+      "learning_rate": 0.0001603034109024281,
+      "loss": 0.6657,
+      "step": 10800
+    },
+    {
+      "epoch": 0.31827605337693815,
+      "grad_norm": 4.991495132446289,
+      "learning_rate": 0.00015954630872793346,
+      "loss": 0.6755,
+      "step": 10900
+    },
+    {
+      "epoch": 0.3211960171693871,
+      "grad_norm": 12.312626838684082,
+      "learning_rate": 0.00015878388083448943,
+      "loss": 0.688,
+      "step": 11000
+    },
+    {
+      "epoch": 0.3241159809618361,
+      "grad_norm": 5.133008003234863,
+      "learning_rate": 0.0001580161954123292,
+      "loss": 0.6165,
+      "step": 11100
+    },
+    {
+      "epoch": 0.32703594475428505,
+      "grad_norm": 7.20357084274292,
+      "learning_rate": 0.00015724332112191028,
+      "loss": 0.6942,
+      "step": 11200
+    },
+    {
+      "epoch": 0.329955908546734,
+      "grad_norm": 7.373988628387451,
+      "learning_rate": 0.00015646532708777356,
+      "loss": 0.6662,
+      "step": 11300
+    },
+    {
+      "epoch": 0.332875872339183,
+      "grad_norm": 5.183440685272217,
+      "learning_rate": 0.00015568228289236102,
+      "loss": 0.6832,
+      "step": 11400
+    },
+    {
+      "epoch": 0.33579583613163194,
+      "grad_norm": 2.848907709121704,
+      "learning_rate": 0.0001548942585697923,
+      "loss": 0.5977,
+      "step": 11500
+    },
+    {
+      "epoch": 0.33871579992408096,
+      "grad_norm": 6.83838415145874,
+      "learning_rate": 0.000154101324599601,
+      "loss": 0.671,
+      "step": 11600
+    },
+    {
+      "epoch": 0.3416357637165299,
+      "grad_norm": 10.248309135437012,
+      "learning_rate": 0.0001533035519004311,
+      "loss": 0.6546,
+      "step": 11700
+    },
+    {
+      "epoch": 0.3445557275089789,
+      "grad_norm": 4.799848556518555,
+      "learning_rate": 0.00015250101182369421,
+      "loss": 0.6847,
+      "step": 11800
+    },
+    {
+      "epoch": 0.34747569130142786,
+      "grad_norm": 7.513195514678955,
+      "learning_rate": 0.0001516937761471879,
+      "loss": 0.6531,
+      "step": 11900
+    },
+    {
+      "epoch": 0.3503956550938768,
+      "grad_norm": 5.581793785095215,
+      "learning_rate": 0.00015088191706867608,
+      "loss": 0.6354,
+      "step": 12000
+    },
+    {
+      "epoch": 0.3533156188863258,
+      "grad_norm": 7.360459804534912,
+      "learning_rate": 0.00015007369358511694,
+      "loss": 0.6267,
+      "step": 12100
+    },
+    {
+      "epoch": 0.3562355826787748,
+      "grad_norm": 8.538688659667969,
+      "learning_rate": 0.0001492528503583914,
+      "loss": 0.7001,
+      "step": 12200
+    },
+    {
+      "epoch": 0.3591555464712238,
+      "grad_norm": 5.245034217834473,
+      "learning_rate": 0.00014842760204184465,
+      "loss": 0.6134,
+      "step": 12300
+    },
+    {
+      "epoch": 0.36207551026367274,
+      "grad_norm": 8.417107582092285,
+      "learning_rate": 0.00014759802244426005,
+      "loss": 0.6335,
+      "step": 12400
+    },
+    {
+      "epoch": 0.3649954740561217,
+      "grad_norm": 3.8830833435058594,
+      "learning_rate": 0.00014676418576180312,
+      "loss": 0.6585,
+      "step": 12500
+    },
+    {
+      "epoch": 0.36791543784857067,
+      "grad_norm": 2.1041855812072754,
+      "learning_rate": 0.00014592616657138586,
+      "loss": 0.591,
+      "step": 12600
+    },
+    {
+      "epoch": 0.37083540164101964,
+      "grad_norm": 6.608900547027588,
+      "learning_rate": 0.00014508403982399636,
+      "loss": 0.649,
+      "step": 12700
+    },
+    {
+      "epoch": 0.3737553654334686,
+      "grad_norm": 3.221425771713257,
+      "learning_rate": 0.0001442378808379957,
+      "loss": 0.6272,
+      "step": 12800
+    },
+    {
+      "epoch": 0.3766753292259176,
+      "grad_norm": 5.6488037109375,
+      "learning_rate": 0.00014338776529238128,
+      "loss": 0.6104,
+      "step": 12900
+    },
+    {
+      "epoch": 0.3795952930183666,
+      "grad_norm": 5.478495121002197,
+      "learning_rate": 0.0001425337692200184,
+      "loss": 0.6142,
+      "step": 13000
+    },
+    {
+      "epoch": 0.38251525681081555,
+      "grad_norm": 5.134850978851318,
+      "learning_rate": 0.00014167596900083998,
+      "loss": 0.6702,
+      "step": 13100
+    },
+    {
+      "epoch": 0.3854352206032645,
+      "grad_norm": 6.333113670349121,
+      "learning_rate": 0.0001408144413550152,
+      "loss": 0.5947,
+      "step": 13200
+    },
+    {
+      "epoch": 0.3883551843957135,
+      "grad_norm": 2.1404473781585693,
+      "learning_rate": 0.00013994926333608793,
+      "loss": 0.6002,
+      "step": 13300
+    },
+    {
+      "epoch": 0.39127514818816245,
+      "grad_norm": 6.199524402618408,
+      "learning_rate": 0.000139080512324085,
+      "loss": 0.6886,
+      "step": 13400
+    },
+    {
+      "epoch": 0.39419511198061147,
+      "grad_norm": 9.652315139770508,
+      "learning_rate": 0.00013820826601859568,
+      "loss": 0.6008,
+      "step": 13500
+    },
+    {
+      "epoch": 0.39711507577306043,
+      "grad_norm": 4.127632141113281,
+      "learning_rate": 0.0001373326024318221,
+      "loss": 0.5349,
+      "step": 13600
+    },
+    {
+      "epoch": 0.4000350395655094,
+      "grad_norm": 5.289997100830078,
+      "learning_rate": 0.00013645359988160232,
+      "loss": 0.5426,
+      "step": 13700
+    },
+    {
+      "epoch": 0.40295500335795836,
+      "grad_norm": 5.238863945007324,
+      "learning_rate": 0.0001355713369844053,
+      "loss": 0.5761,
+      "step": 13800
+    },
+    {
+      "epoch": 0.40587496715040733,
+      "grad_norm": 2.4321208000183105,
+      "learning_rate": 0.0001346858926483,
+      "loss": 0.5926,
+      "step": 13900
+    },
+    {
+      "epoch": 0.4087949309428563,
+      "grad_norm": 8.464098930358887,
+      "learning_rate": 0.0001337973460658976,
+      "loss": 0.5804,
+      "step": 14000
+    },
+    {
+      "epoch": 0.41171489473530526,
+      "grad_norm": 3.87615704536438,
+      "learning_rate": 0.000132905776707269,
+      "loss": 0.5995,
+      "step": 14100
+    },
+    {
+      "epoch": 0.4146348585277543,
+      "grad_norm": 6.268438339233398,
+      "learning_rate": 0.0001320112643128368,
+      "loss": 0.5306,
+      "step": 14200
+    },
+    {
+      "epoch": 0.41755482232020324,
+      "grad_norm": 3.1259851455688477,
+      "learning_rate": 0.00013111388888624386,
+      "loss": 0.591,
+      "step": 14300
+    },
+    {
+      "epoch": 0.4204747861126522,
+      "grad_norm": 5.6857829093933105,
+      "learning_rate": 0.00013021373068719742,
+      "loss": 0.6046,
+      "step": 14400
+    },
+    {
+      "epoch": 0.4233947499051012,
+      "grad_norm": 6.713433742523193,
+      "learning_rate": 0.00012931087022429133,
+      "loss": 0.6183,
+      "step": 14500
+    },
+    {
+      "epoch": 0.42631471369755014,
+      "grad_norm": 3.385892152786255,
+      "learning_rate": 0.00012841445577833958,
+      "loss": 0.578,
+      "step": 14600
+    },
+    {
+      "epoch": 0.4292346774899991,
+      "grad_norm": 9.377425193786621,
+      "learning_rate": 0.00012750645827667818,
+      "loss": 0.6234,
+      "step": 14700
+    },
+    {
+      "epoch": 0.43215464128244807,
+      "grad_norm": 2.394103765487671,
+      "learning_rate": 0.00012659600064492303,
+      "loss": 0.5021,
+      "step": 14800
+    },
+    {
+      "epoch": 0.4350746050748971,
+      "grad_norm": 7.460124492645264,
+      "learning_rate": 0.00012568316431283124,
+      "loss": 0.5953,
+      "step": 14900
+    },
+    {
+      "epoch": 0.43799456886734606,
+      "grad_norm": 3.1461405754089355,
+      "learning_rate": 0.00012476803092290696,
+      "loss": 0.5579,
+      "step": 15000
+    },
+    {
+      "epoch": 0.440914532659795,
+      "grad_norm": 6.676076412200928,
+      "learning_rate": 0.00012385068232309906,
+      "loss": 0.6248,
+      "step": 15100
+    },
+    {
+      "epoch": 0.443834496452244,
+      "grad_norm": 4.148955821990967,
+      "learning_rate": 0.00012293120055948103,
+      "loss": 0.5219,
+      "step": 15200
+    },
+    {
+      "epoch": 0.44675446024469295,
+      "grad_norm": 2.290743589401245,
+      "learning_rate": 0.00012200966786891287,
+      "loss": 0.5686,
+      "step": 15300
+    },
+    {
+      "epoch": 0.4496744240371419,
+      "grad_norm": 6.50786828994751,
+      "learning_rate": 0.00012108616667168586,
+      "loss": 0.5667,
+      "step": 15400
+    },
+    {
+      "epoch": 0.45259438782959094,
+      "grad_norm": 4.148983001708984,
+      "learning_rate": 0.00012016077956415119,
+      "loss": 0.5426,
+      "step": 15500
+    },
+    {
+      "epoch": 0.4555143516220399,
+      "grad_norm": 4.001679420471191,
+      "learning_rate": 0.00011923358931133255,
+      "loss": 0.568,
+      "step": 15600
+    },
+    {
+      "epoch": 0.45843431541448887,
+      "grad_norm": 3.7630743980407715,
+      "learning_rate": 0.00011830467883952387,
+      "loss": 0.5927,
+      "step": 15700
+    },
+    {
+      "epoch": 0.46135427920693783,
+      "grad_norm": 7.692181587219238,
+      "learning_rate": 0.00011737413122887249,
+      "loss": 0.6512,
+      "step": 15800
+    },
+    {
+      "epoch": 0.4642742429993868,
+      "grad_norm": 1.6598546504974365,
+      "learning_rate": 0.00011644202970594859,
+      "loss": 0.5219,
+      "step": 15900
+    },
+    {
+      "epoch": 0.46719420679183576,
+      "grad_norm": 6.290797710418701,
+      "learning_rate": 0.00011550845763630155,
+      "loss": 0.6026,
+      "step": 16000
+    },
+    {
+      "epoch": 0.47011417058428473,
+      "grad_norm": 7.756636619567871,
+      "learning_rate": 0.00011457349851700407,
+      "loss": 0.619,
+      "step": 16100
+    },
+    {
+      "epoch": 0.47303413437673375,
+      "grad_norm": 7.793608665466309,
+      "learning_rate": 0.00011363723596918397,
+      "loss": 0.634,
+      "step": 16200
+    },
+    {
+      "epoch": 0.4759540981691827,
+      "grad_norm": 3.12341046333313,
+      "learning_rate": 0.00011269975373054557,
+      "loss": 0.5714,
+      "step": 16300
+    },
+    {
+      "epoch": 0.4788740619616317,
+      "grad_norm": 6.63306999206543,
+      "learning_rate": 0.00011176113564788022,
+      "loss": 0.5377,
+      "step": 16400
+    },
+    {
+      "epoch": 0.48179402575408065,
+      "grad_norm": 3.5353915691375732,
+      "learning_rate": 0.0001108214656695672,
+      "loss": 0.5573,
+      "step": 16500
+    },
+    {
+      "epoch": 0.4847139895465296,
+      "grad_norm": 5.9525933265686035,
+      "learning_rate": 0.00010988082783806548,
+      "loss": 0.5805,
+      "step": 16600
+    },
+    {
+      "epoch": 0.4876339533389786,
+      "grad_norm": 5.2525410652160645,
+      "learning_rate": 0.00010893930628239719,
+      "loss": 0.5826,
+      "step": 16700
+    },
+    {
+      "epoch": 0.4905539171314276,
+      "grad_norm": 6.823146343231201,
+      "learning_rate": 0.00010799698521062318,
+      "loss": 0.5534,
+      "step": 16800
+    },
+    {
+      "epoch": 0.49347388092387656,
+      "grad_norm": 4.929098606109619,
+      "learning_rate": 0.00010705394890231171,
+      "loss": 0.5175,
+      "step": 16900
+    },
+    {
+      "epoch": 0.4963938447163255,
+      "grad_norm": 3.4094526767730713,
+      "learning_rate": 0.0001061102817010005,
+      "loss": 0.5499,
+      "step": 17000
+    },
+    {
+      "epoch": 0.4993138085087745,
+      "grad_norm": 3.1009860038757324,
+      "learning_rate": 0.00010516606800665324,
+      "loss": 0.5971,
+      "step": 17100
+    },
+    {
+      "epoch": 0.5022337723012235,
+      "grad_norm": 1.6674872636795044,
+      "learning_rate": 0.00010422139226811105,
+      "loss": 0.5571,
+      "step": 17200
+    },
+    {
+      "epoch": 0.5051537360936724,
+      "grad_norm": 5.221986293792725,
+      "learning_rate": 0.0001032763389755395,
+      "loss": 0.6719,
+      "step": 17300
+    },
+    {
+      "epoch": 0.5080736998861214,
+      "grad_norm": 5.639619827270508,
+      "learning_rate": 0.00010233099265287182,
+      "loss": 0.5489,
+      "step": 17400
+    },
+    {
+      "epoch": 0.5109936636785704,
+      "grad_norm": 5.451557636260986,
+      "learning_rate": 0.0001013854378502496,
+      "loss": 0.6173,
+      "step": 17500
+    },
+    {
+      "epoch": 0.5139136274710193,
+      "grad_norm": 5.069969177246094,
+      "learning_rate": 0.00010043975913646017,
+      "loss": 0.5701,
+      "step": 17600
+    },
+    {
+      "epoch": 0.5168335912634683,
+      "grad_norm": 1.3468672037124634,
+      "learning_rate": 9.949404109137361e-05,
+      "loss": 0.5582,
+      "step": 17700
+    },
+    {
+      "epoch": 0.5197535550559174,
+      "grad_norm": 7.0001420974731445,
+      "learning_rate": 9.854836829837741e-05,
+      "loss": 0.6168,
+      "step": 17800
+    },
+    {
+      "epoch": 0.5226735188483663,
+      "grad_norm": 2.2107551097869873,
+      "learning_rate": 9.760282533681206e-05,
+      "loss": 0.4888,
+      "step": 17900
+    },
+    {
+      "epoch": 0.5255934826408153,
+      "grad_norm": 5.56157922744751,
+      "learning_rate": 9.665749677440597e-05,
+      "loss": 0.5833,
+      "step": 18000
+    },
+    {
+      "epoch": 0.5285134464332643,
+      "grad_norm": 3.0002171993255615,
+      "learning_rate": 9.571246715971211e-05,
+      "loss": 0.5705,
+      "step": 18100
+    },
+    {
+      "epoch": 0.5314334102257132,
+      "grad_norm": 3.6839287281036377,
+      "learning_rate": 9.476782101454611e-05,
+      "loss": 0.5261,
+      "step": 18200
+    },
+    {
+      "epoch": 0.5343533740181622,
+      "grad_norm": 6.553487777709961,
+      "learning_rate": 9.382364282642675e-05,
+      "loss": 0.5936,
+      "step": 18300
+    },
+    {
+      "epoch": 0.5372733378106112,
+      "grad_norm": 6.64651346206665,
+      "learning_rate": 9.28800170410195e-05,
+      "loss": 0.5576,
+      "step": 18400
+    },
+    {
+      "epoch": 0.5401933016030601,
+      "grad_norm": 4.049524307250977,
+      "learning_rate": 9.193702805458388e-05,
+      "loss": 0.5326,
+      "step": 18500
+    },
+    {
+      "epoch": 0.5431132653955091,
+      "grad_norm": 3.995262861251831,
+      "learning_rate": 9.09947602064253e-05,
+      "loss": 0.577,
+      "step": 18600
+    },
+    {
+      "epoch": 0.546033229187958,
+      "grad_norm": 5.921045303344727,
+      "learning_rate": 9.005329777135172e-05,
+      "loss": 0.5901,
+      "step": 18700
+    },
+    {
+      "epoch": 0.548953192980407,
+      "grad_norm": 1.7855268716812134,
+      "learning_rate": 8.911272495213639e-05,
+      "loss": 0.5026,
+      "step": 18800
+    },
+    {
+      "epoch": 0.551873156772856,
+      "grad_norm": 2.2023262977600098,
+      "learning_rate": 8.818251676677135e-05,
+      "loss": 0.526,
+      "step": 18900
+    },
+    {
+      "epoch": 0.5547931205653049,
+      "grad_norm": 2.3260550498962402,
+      "learning_rate": 8.724396446837928e-05,
+      "loss": 0.5715,
+      "step": 19000
+    },
+    {
+      "epoch": 0.557713084357754,
+      "grad_norm": 4.143796920776367,
+      "learning_rate": 8.63065530477612e-05,
+      "loss": 0.5346,
+      "step": 19100
+    },
+    {
+      "epoch": 0.560633048150203,
+      "grad_norm": 3.3329811096191406,
+      "learning_rate": 8.537036634537457e-05,
+      "loss": 0.525,
+      "step": 19200
+    },
+    {
+      "epoch": 0.563553011942652,
+      "grad_norm": 2.1307923793792725,
+      "learning_rate": 8.443548809214026e-05,
+      "loss": 0.6003,
+      "step": 19300
+    },
+    {
+      "epoch": 0.5664729757351009,
+      "grad_norm": 9.149153709411621,
+      "learning_rate": 8.350200190195363e-05,
+      "loss": 0.5832,
+      "step": 19400
+    },
+    {
+      "epoch": 0.5693929395275499,
+      "grad_norm": 2.979999303817749,
+      "learning_rate": 8.25699912642064e-05,
+      "loss": 0.5697,
+      "step": 19500
+    },
+    {
+      "epoch": 0.5723129033199988,
+      "grad_norm": 3.4557132720947266,
+      "learning_rate": 8.163953953631926e-05,
+      "loss": 0.5883,
+      "step": 19600
+    },
+    {
+      "epoch": 0.5752328671124478,
+      "grad_norm": 5.705073833465576,
+      "learning_rate": 8.071072993628694e-05,
+      "loss": 0.5408,
+      "step": 19700
+    },
+    {
+      "epoch": 0.5781528309048968,
+      "grad_norm": 4.233023166656494,
+      "learning_rate": 7.978364553523492e-05,
+      "loss": 0.564,
+      "step": 19800
+    },
+    {
+      "epoch": 0.5810727946973457,
+      "grad_norm": 4.051882266998291,
+      "learning_rate": 7.885836924998998e-05,
+      "loss": 0.5367,
+      "step": 19900
+    },
+    {
+      "epoch": 0.5839927584897947,
+      "grad_norm": 6.308223724365234,
+      "learning_rate": 7.793498383566398e-05,
+      "loss": 0.5494,
+      "step": 20000
+    },
+    {
+      "epoch": 0.5869127222822437,
+      "grad_norm": 3.083712577819824,
+      "learning_rate": 7.701357187825272e-05,
+      "loss": 0.5712,
+      "step": 20100
+    },
+    {
+      "epoch": 0.5898326860746926,
+      "grad_norm": 4.624127388000488,
+      "learning_rate": 7.609421578724932e-05,
+      "loss": 0.6324,
+      "step": 20200
+    },
+    {
+      "epoch": 0.5927526498671416,
+      "grad_norm": 2.8166234493255615,
+      "learning_rate": 7.517699778827388e-05,
+      "loss": 0.5447,
+      "step": 20300
+    },
+    {
+      "epoch": 0.5956726136595906,
+      "grad_norm": 5.028948783874512,
+      "learning_rate": 7.42619999157191e-05,
+      "loss": 0.5499,
+      "step": 20400
+    },
+    {
+      "epoch": 0.5985925774520396,
+      "grad_norm": 2.2335968017578125,
+      "learning_rate": 7.334930400541347e-05,
+      "loss": 0.5178,
+      "step": 20500
+    },
+    {
+      "epoch": 0.6015125412444886,
+      "grad_norm": 2.491394281387329,
+      "learning_rate": 7.243899168730202e-05,
+      "loss": 0.5571,
+      "step": 20600
+    },
+    {
+      "epoch": 0.6044325050369376,
+      "grad_norm": 5.571224212646484,
+      "learning_rate": 7.153114437814525e-05,
+      "loss": 0.5365,
+      "step": 20700
+    },
+    {
+      "epoch": 0.6073524688293865,
+      "grad_norm": 3.0990610122680664,
+      "learning_rate": 7.062584327423772e-05,
+      "loss": 0.505,
+      "step": 20800
+    },
+    {
+      "epoch": 0.6102724326218355,
+      "grad_norm": 2.507359266281128,
+      "learning_rate": 6.972316934414562e-05,
+      "loss": 0.5801,
+      "step": 20900
+    },
+    {
+      "epoch": 0.6131923964142845,
+      "grad_norm": 2.0391249656677246,
+      "learning_rate": 6.882320332146542e-05,
+      "loss": 0.4986,
+      "step": 21000
+    },
+    {
+      "epoch": 0.6161123602067334,
+      "grad_norm": 6.61077356338501,
+      "learning_rate": 6.793498340758233e-05,
+      "loss": 0.5985,
+      "step": 21100
+    },
+    {
+      "epoch": 0.6190323239991824,
+      "grad_norm": 2.313990831375122,
+      "learning_rate": 6.704064534178382e-05,
+      "loss": 0.5378,
+      "step": 21200
+    },
+    {
+      "epoch": 0.6219522877916314,
+      "grad_norm": 3.221607208251953,
+      "learning_rate": 6.614925510370827e-05,
+      "loss": 0.5767,
+      "step": 21300
+    },
+    {
+      "epoch": 0.6248722515840803,
+      "grad_norm": 2.1119728088378906,
+      "learning_rate": 6.526089241775799e-05,
+      "loss": 0.5344,
+      "step": 21400
+    },
+    {
+      "epoch": 0.6277922153765293,
+      "grad_norm": 6.011439323425293,
+      "learning_rate": 6.437563673755653e-05,
+      "loss": 0.6197,
+      "step": 21500
+    },
+    {
+      "epoch": 0.6307121791689783,
+      "grad_norm": 1.3864915370941162,
+      "learning_rate": 6.349356723884206e-05,
+      "loss": 0.5489,
+      "step": 21600
+    },
+    {
+      "epoch": 0.6336321429614272,
+      "grad_norm": 2.4754810333251953,
+      "learning_rate": 6.261476281238625e-05,
+      "loss": 0.4908,
+      "step": 21700
+    },
+    {
+      "epoch": 0.6365521067538763,
+      "grad_norm": 3.065643787384033,
+      "learning_rate": 6.173930205693829e-05,
+      "loss": 0.5829,
+      "step": 21800
+    },
+    {
+      "epoch": 0.6394720705463253,
+      "grad_norm": 4.440421104431152,
+      "learning_rate": 6.086726327219525e-05,
+      "loss": 0.5175,
+      "step": 21900
+    },
+    {
+      "epoch": 0.6423920343387742,
+      "grad_norm": 2.5433878898620605,
+      "learning_rate": 5.999872445179904e-05,
+      "loss": 0.604,
+      "step": 22000
+    },
+    {
+      "epoch": 0.6453119981312232,
+      "grad_norm": 5.749415397644043,
+      "learning_rate": 5.9133763276360855e-05,
+      "loss": 0.5274,
+      "step": 22100
+    },
+    {
+      "epoch": 0.6482319619236722,
+      "grad_norm": 4.130051136016846,
+      "learning_rate": 5.8272457106513435e-05,
+      "loss": 0.524,
+      "step": 22200
+    },
+    {
+      "epoch": 0.6511519257161211,
+      "grad_norm": 4.023646831512451,
+      "learning_rate": 5.741488297599216e-05,
+      "loss": 0.5641,
+      "step": 22300
+    },
+    {
+      "epoch": 0.6540718895085701,
+      "grad_norm": 3.071547746658325,
+      "learning_rate": 5.656111758474528e-05,
+      "loss": 0.5693,
+      "step": 22400
+    },
+    {
+      "epoch": 0.6569918533010191,
+      "grad_norm": 2.316500663757324,
+      "learning_rate": 5.571123729207408e-05,
+      "loss": 0.4671,
+      "step": 22500
+    },
+    {
+      "epoch": 0.659911817093468,
+      "grad_norm": 2.743230104446411,
+      "learning_rate": 5.4865318109803064e-05,
+      "loss": 0.5422,
+      "step": 22600
+    },
+    {
+      "epoch": 0.662831780885917,
+      "grad_norm": 4.95118522644043,
+      "learning_rate": 5.402343569548214e-05,
+      "loss": 0.5305,
+      "step": 22700
+    },
+    {
+      "epoch": 0.665751744678366,
+      "grad_norm": 5.621161460876465,
+      "learning_rate": 5.318566534561956e-05,
+      "loss": 0.5706,
+      "step": 22800
+    },
+    {
+      "epoch": 0.6686717084708149,
+      "grad_norm": 6.144950866699219,
+      "learning_rate": 5.236039685178543e-05,
+      "loss": 0.5763,
+      "step": 22900
+    },
+    {
+      "epoch": 0.6715916722632639,
+      "grad_norm": 2.8691632747650146,
+      "learning_rate": 5.1531032059287074e-05,
+      "loss": 0.5099,
+      "step": 23000
+    },
+    {
+      "epoch": 0.674511636055713,
+      "grad_norm": 1.8906828165054321,
+      "learning_rate": 5.070600224752108e-05,
+      "loss": 0.5564,
+      "step": 23100
+    },
+    {
+      "epoch": 0.6774315998481619,
+      "grad_norm": 2.1284193992614746,
+      "learning_rate": 4.9885381205728124e-05,
+      "loss": 0.5256,
+      "step": 23200
+    },
+    {
+      "epoch": 0.6803515636406109,
+      "grad_norm": 8.401644706726074,
+      "learning_rate": 4.9069242328835974e-05,
+      "loss": 0.5609,
+      "step": 23300
+    },
+    {
+      "epoch": 0.6832715274330599,
+      "grad_norm": 3.2185475826263428,
+      "learning_rate": 4.8257658610895485e-05,
+      "loss": 0.5466,
+      "step": 23400
+    },
+    {
+      "epoch": 0.6861914912255088,
+      "grad_norm": 1.7117905616760254,
+      "learning_rate": 4.7450702638551795e-05,
+      "loss": 0.532,
+      "step": 23500
+    },
+    {
+      "epoch": 0.6891114550179578,
+      "grad_norm": 5.279407501220703,
+      "learning_rate": 4.664844658455262e-05,
+      "loss": 0.4883,
+      "step": 23600
+    },
+    {
+      "epoch": 0.6920314188104068,
+      "grad_norm": 3.187605619430542,
+      "learning_rate": 4.5850962201292956e-05,
+      "loss": 0.5377,
+      "step": 23700
+    },
+    {
+      "epoch": 0.6949513826028557,
+      "grad_norm": 3.679347515106201,
+      "learning_rate": 4.5058320814397984e-05,
+      "loss": 0.5999,
+      "step": 23800
+    },
+    {
+      "epoch": 0.6978713463953047,
+      "grad_norm": 2.252751588821411,
+      "learning_rate": 4.427059331634348e-05,
+      "loss": 0.479,
+      "step": 23900
+    },
+    {
+      "epoch": 0.7007913101877536,
+      "grad_norm": 4.126784801483154,
+      "learning_rate": 4.348785016011566e-05,
+      "loss": 0.5708,
+      "step": 24000
+    },
+    {
+      "epoch": 0.7037112739802026,
+      "grad_norm": 2.982395648956299,
+      "learning_rate": 4.2710161352909674e-05,
+      "loss": 0.4497,
+      "step": 24100
+    },
+    {
+      "epoch": 0.7066312377726516,
+      "grad_norm": 2.803725242614746,
+      "learning_rate": 4.193759644986852e-05,
+      "loss": 0.526,
+      "step": 24200
+    },
+    {
+      "epoch": 0.7095512015651005,
+      "grad_norm": 2.7953624725341797,
+      "learning_rate": 4.117022454786209e-05,
+      "loss": 0.5743,
+      "step": 24300
+    },
+    {
+      "epoch": 0.7124711653575496,
+      "grad_norm": 3.2696502208709717,
+      "learning_rate": 4.040811427930715e-05,
+      "loss": 0.531,
+      "step": 24400
+    },
+    {
+      "epoch": 0.7153911291499986,
+      "grad_norm": 5.39483118057251,
+      "learning_rate": 3.9651333806029066e-05,
+      "loss": 0.6086,
+      "step": 24500
+    },
+    {
+      "epoch": 0.7183110929424475,
+      "grad_norm": 2.236680030822754,
+      "learning_rate": 3.8899950813165595e-05,
+      "loss": 0.538,
+      "step": 24600
+    },
+    {
+      "epoch": 0.7212310567348965,
+      "grad_norm": 5.033971786499023,
+      "learning_rate": 3.815403250311317e-05,
+      "loss": 0.5001,
+      "step": 24700
+    },
+    {
+      "epoch": 0.7241510205273455,
+      "grad_norm": 3.3246283531188965,
+      "learning_rate": 3.741364558951638e-05,
+      "loss": 0.5278,
+      "step": 24800
+    },
+    {
+      "epoch": 0.7270709843197944,
+      "grad_norm": 1.9075288772583008,
+      "learning_rate": 3.667885629130124e-05,
+      "loss": 0.5903,
+      "step": 24900
+    },
+    {
+      "epoch": 0.7299909481122434,
+      "grad_norm": 3.299224853515625,
+      "learning_rate": 3.5949730326752854e-05,
+      "loss": 0.4959,
+      "step": 25000
+    },
+    {
+      "epoch": 0.7329109119046924,
+      "grad_norm": 3.502321720123291,
+      "learning_rate": 3.522633290763746e-05,
+      "loss": 0.5437,
+      "step": 25100
+    },
+    {
+      "epoch": 0.7358308756971413,
+      "grad_norm": 6.021385192871094,
+      "learning_rate": 3.45087287333701e-05,
+      "loss": 0.5548,
+      "step": 25200
+    },
+    {
+      "epoch": 0.7387508394895903,
+      "grad_norm": 2.9654223918914795,
+      "learning_rate": 3.379698198522798e-05,
+      "loss": 0.5051,
+      "step": 25300
+    },
+    {
+      "epoch": 0.7416708032820393,
+      "grad_norm": 5.164859294891357,
+      "learning_rate": 3.30911563206103e-05,
+      "loss": 0.5165,
+      "step": 25400
+    },
+    {
+      "epoch": 0.7445907670744882,
+      "grad_norm": 3.9957275390625,
+      "learning_rate": 3.239131486734478e-05,
+      "loss": 0.4851,
+      "step": 25500
+    },
+    {
+      "epoch": 0.7475107308669372,
+      "grad_norm": 4.626624584197998,
+      "learning_rate": 3.1697520218041555e-05,
+      "loss": 0.5368,
+      "step": 25600
+    },
+    {
+      "epoch": 0.7504306946593863,
+      "grad_norm": 2.424008846282959,
+      "learning_rate": 3.100983442449506e-05,
+      "loss": 0.4919,
+      "step": 25700
+    },
+    {
+      "epoch": 0.7533506584518352,
+      "grad_norm": 2.843548536300659,
+      "learning_rate": 3.032831899213433e-05,
+      "loss": 0.5479,
+      "step": 25800
+    },
+    {
+      "epoch": 0.7562706222442842,
+      "grad_norm": 2.585984945297241,
+      "learning_rate": 2.965303487452189e-05,
+      "loss": 0.4914,
+      "step": 25900
+    },
+    {
+      "epoch": 0.7591905860367332,
+      "grad_norm": 2.7971699237823486,
+      "learning_rate": 2.8984042467902207e-05,
+      "loss": 0.5433,
+      "step": 26000
+    },
+    {
+      "epoch": 0.7621105498291821,
+      "grad_norm": 2.5568065643310547,
+      "learning_rate": 2.832140160580007e-05,
+      "loss": 0.5143,
+      "step": 26100
+    },
+    {
+      "epoch": 0.7650305136216311,
+      "grad_norm": 2.1987948417663574,
+      "learning_rate": 2.7665171553669023e-05,
+      "loss": 0.47,
+      "step": 26200
+    },
+    {
+      "epoch": 0.7679504774140801,
+      "grad_norm": 2.724154233932495,
+      "learning_rate": 2.701541100359093e-05,
+      "loss": 0.4984,
+      "step": 26300
+    },
+    {
+      "epoch": 0.770870441206529,
+      "grad_norm": 3.9086155891418457,
+      "learning_rate": 2.6372178069026464e-05,
+      "loss": 0.5805,
+      "step": 26400
+    },
+    {
+      "epoch": 0.773790404998978,
+      "grad_norm": 3.4147980213165283,
+      "learning_rate": 2.573553027961778e-05,
+      "loss": 0.4967,
+      "step": 26500
+    },
+    {
+      "epoch": 0.776710368791427,
+      "grad_norm": 6.612050533294678,
+      "learning_rate": 2.5105524576042894e-05,
+      "loss": 0.5695,
+      "step": 26600
+    },
+    {
+      "epoch": 0.7796303325838759,
+      "grad_norm": 3.5038676261901855,
+      "learning_rate": 2.4482217304923273e-05,
+      "loss": 0.5575,
+      "step": 26700
+    },
+    {
+      "epoch": 0.7825502963763249,
+      "grad_norm": 1.891401767730713,
+      "learning_rate": 2.3865664213784024e-05,
+      "loss": 0.5505,
+      "step": 26800
+    },
+    {
+      "epoch": 0.7854702601687739,
+      "grad_norm": 2.7758102416992188,
+      "learning_rate": 2.32559204460682e-05,
+      "loss": 0.548,
+      "step": 26900
+    },
+    {
+      "epoch": 0.7883902239612229,
+      "grad_norm": 8.392036437988281,
+      "learning_rate": 2.2653040536204616e-05,
+      "loss": 0.5254,
+      "step": 27000
+    },
+    {
+      "epoch": 0.7913101877536719,
+      "grad_norm": 1.787404179573059,
+      "learning_rate": 2.2057078404730637e-05,
+      "loss": 0.5049,
+      "step": 27100
+    },
+    {
+      "epoch": 0.7942301515461209,
+      "grad_norm": 4.141063213348389,
+      "learning_rate": 2.1468087353469336e-05,
+      "loss": 0.5384,
+      "step": 27200
+    },
+    {
+      "epoch": 0.7971501153385698,
+      "grad_norm": 4.328978538513184,
+      "learning_rate": 2.0886120060762516e-05,
+      "loss": 0.5508,
+      "step": 27300
+    },
+    {
+      "epoch": 0.8000700791310188,
+      "grad_norm": 2.4056620597839355,
+      "learning_rate": 2.0311228576759157e-05,
+      "loss": 0.5034,
+      "step": 27400
+    },
+    {
+      "epoch": 0.8029900429234678,
+      "grad_norm": 2.209656000137329,
+      "learning_rate": 1.9743464318760087e-05,
+      "loss": 0.585,
+      "step": 27500
+    },
+    {
+      "epoch": 0.8059100067159167,
+      "grad_norm": 3.6636762619018555,
+      "learning_rate": 1.9182878066619337e-05,
+      "loss": 0.5427,
+      "step": 27600
+    },
+    {
+      "epoch": 0.8088299705083657,
+      "grad_norm": 3.2900936603546143,
+      "learning_rate": 1.862951995820257e-05,
+      "loss": 0.546,
+      "step": 27700
+    },
+    {
+      "epoch": 0.8117499343008147,
+      "grad_norm": 1.6237975358963013,
+      "learning_rate": 1.8083439484902754e-05,
+      "loss": 0.51,
+      "step": 27800
+    },
+    {
+      "epoch": 0.8146698980932636,
+      "grad_norm": 4.96632719039917,
+      "learning_rate": 1.7544685487213718e-05,
+      "loss": 0.5784,
+      "step": 27900
+    },
+    {
+      "epoch": 0.8175898618857126,
+      "grad_norm": 4.221993446350098,
+      "learning_rate": 1.701330615036194e-05,
+      "loss": 0.5209,
+      "step": 28000
+    },
+    {
+      "epoch": 0.8205098256781616,
+      "grad_norm": 3.626847982406616,
+      "learning_rate": 1.6489348999997068e-05,
+      "loss": 0.4913,
+      "step": 28100
+    },
+    {
+      "epoch": 0.8234297894706105,
+      "grad_norm": 4.361175537109375,
+      "learning_rate": 1.5972860897941167e-05,
+      "loss": 0.5247,
+      "step": 28200
+    },
+    {
+      "epoch": 0.8263497532630595,
+      "grad_norm": 4.415472507476807,
+      "learning_rate": 1.5463888037997522e-05,
+      "loss": 0.5272,
+      "step": 28300
+    },
+    {
+      "epoch": 0.8292697170555086,
+      "grad_norm": 3.3726589679718018,
+      "learning_rate": 1.4962475941819132e-05,
+      "loss": 0.474,
+      "step": 28400
+    },
+    {
+      "epoch": 0.8321896808479575,
+      "grad_norm": 1.9652336835861206,
+      "learning_rate": 1.4468669454837402e-05,
+      "loss": 0.5134,
+      "step": 28500
+    },
+    {
+      "epoch": 0.8351096446404065,
+      "grad_norm": 2.4251391887664795,
+      "learning_rate": 1.3982512742251197e-05,
+      "loss": 0.5433,
+      "step": 28600
+    },
+    {
+      "epoch": 0.8380296084328555,
+      "grad_norm": 1.2398502826690674,
+      "learning_rate": 1.3504049285076725e-05,
+      "loss": 0.5545,
+      "step": 28700
+    },
+    {
+      "epoch": 0.8409495722253044,
+      "grad_norm": 3.1720731258392334,
+      "learning_rate": 1.3033321876258742e-05,
+      "loss": 0.5603,
+      "step": 28800
+    },
+    {
+      "epoch": 0.8438695360177534,
+      "grad_norm": 2.1568796634674072,
+      "learning_rate": 1.2570372616843229e-05,
+      "loss": 0.472,
+      "step": 28900
+    },
+    {
+      "epoch": 0.8467894998102024,
+      "grad_norm": 2.8056724071502686,
+      "learning_rate": 1.211975536822222e-05,
+      "loss": 0.5052,
+      "step": 29000
+    },
+    {
+      "epoch": 0.8497094636026513,
+      "grad_norm": 1.3424701690673828,
+      "learning_rate": 1.1672407122591966e-05,
+      "loss": 0.4589,
+      "step": 29100
+    },
+    {
+      "epoch": 0.8526294273951003,
+      "grad_norm": 2.5995755195617676,
+      "learning_rate": 1.123295874422794e-05,
+      "loss": 0.5132,
+      "step": 29200
+    },
+    {
+      "epoch": 0.8555493911875492,
+      "grad_norm": 3.993947744369507,
+      "learning_rate": 1.0801449536633657e-05,
+      "loss": 0.4967,
+      "step": 29300
+    },
+    {
+      "epoch": 0.8584693549799982,
+      "grad_norm": 1.6350364685058594,
+      "learning_rate": 1.0377918093246986e-05,
+      "loss": 0.4799,
+      "step": 29400
+    },
+    {
+      "epoch": 0.8613893187724472,
+      "grad_norm": 3.297436237335205,
+      "learning_rate": 9.96240229398826e-06,
+      "loss": 0.4838,
+      "step": 29500
+    },
+    {
+      "epoch": 0.8643092825648961,
+      "grad_norm": 2.4876105785369873,
+      "learning_rate": 9.554939301872545e-06,
+      "loss": 0.4792,
+      "step": 29600
+    },
+    {
+      "epoch": 0.8672292463573452,
+      "grad_norm": 3.8208444118499756,
+      "learning_rate": 9.155565559685697e-06,
+      "loss": 0.5394,
+      "step": 29700
+    },
+    {
+      "epoch": 0.8701492101497942,
+      "grad_norm": 1.8198366165161133,
+      "learning_rate": 8.764316786725013e-06,
+      "loss": 0.4818,
+      "step": 29800
+    },
+    {
+      "epoch": 0.8730691739422431,
+      "grad_norm": 4.755416393280029,
+      "learning_rate": 8.381227975604677e-06,
+      "loss": 0.4975,
+      "step": 29900
+    },
+    {
+      "epoch": 0.8759891377346921,
+      "grad_norm": 2.5053181648254395,
+      "learning_rate": 8.006333389125908e-06,
+      "loss": 0.4599,
+      "step": 30000
+    },
+    {
+      "epoch": 0.8789091015271411,
+      "grad_norm": NaN,
+      "learning_rate": 7.643292389855882e-06,
+      "loss": 0.5384,
+      "step": 30100
+    },
+    {
+      "epoch": 0.88182906531959,
+      "grad_norm": 5.616009712219238,
+      "learning_rate": 7.28480334116276e-06,
+      "loss": 0.5446,
+      "step": 30200
+    },
+    {
+      "epoch": 0.884749029112039,
+      "grad_norm": 2.7977347373962402,
+      "learning_rate": 6.934606579435754e-06,
+      "loss": 0.5179,
+      "step": 30300
+    },
+    {
+      "epoch": 0.887668992904488,
+      "grad_norm": 2.461843490600586,
+      "learning_rate": 6.5927334256680735e-06,
+      "loss": 0.5078,
+      "step": 30400
+    },
+    {
+      "epoch": 0.8905889566969369,
+      "grad_norm": 5.445867538452148,
+      "learning_rate": 6.259214456403772e-06,
+      "loss": 0.6466,
+      "step": 30500
+    },
+    {
+      "epoch": 0.8935089204893859,
+      "grad_norm": 2.652655601501465,
+      "learning_rate": 5.934079501003087e-06,
+      "loss": 0.5021,
+      "step": 30600
+    },
+    {
+      "epoch": 0.8964288842818349,
+      "grad_norm": 2.0053634643554688,
+      "learning_rate": 5.617357638974574e-06,
+      "loss": 0.4928,
+      "step": 30700
+    },
+    {
+      "epoch": 0.8993488480742838,
+      "grad_norm": 2.91902232170105,
+      "learning_rate": 5.309077197374246e-06,
+      "loss": 0.5492,
+      "step": 30800
+    },
+    {
+      "epoch": 0.9022688118667328,
+      "grad_norm": 2.4080049991607666,
+      "learning_rate": 5.009265748272041e-06,
+      "loss": 0.4833,
+      "step": 30900
+    },
+    {
+      "epoch": 0.9051887756591819,
+      "grad_norm": 2.9708127975463867,
+      "learning_rate": 4.717950106285862e-06,
+      "loss": 0.5559,
+      "step": 31000
+    },
+    {
+      "epoch": 0.9081087394516308,
+      "grad_norm": 1.8519601821899414,
+      "learning_rate": 4.4351563261832984e-06,
+      "loss": 0.5401,
+      "step": 31100
+    },
+    {
+      "epoch": 0.9110287032440798,
+      "grad_norm": 4.341070652008057,
+      "learning_rate": 4.160909700551341e-06,
+      "loss": 0.4997,
+      "step": 31200
+    },
+    {
+      "epoch": 0.9139486670365288,
+      "grad_norm": 2.3663878440856934,
+      "learning_rate": 3.895234757534249e-06,
+      "loss": 0.4632,
+      "step": 31300
+    },
+    {
+      "epoch": 0.9168686308289777,
+      "grad_norm": 7.448829174041748,
+      "learning_rate": 3.6381552586398193e-06,
+      "loss": 0.4853,
+      "step": 31400
+    },
+    {
+      "epoch": 0.9197885946214267,
+      "grad_norm": 5.03635311126709,
+      "learning_rate": 3.392136072366148e-06,
+      "loss": 0.57,
+      "step": 31500
+    },
+    {
+      "epoch": 0.9227085584138757,
+      "grad_norm": 1.9699945449829102,
+      "learning_rate": 3.1522291550866503e-06,
+      "loss": 0.5089,
+      "step": 31600
+    },
+    {
+      "epoch": 0.9256285222063246,
+      "grad_norm": 2.2074334621429443,
+      "learning_rate": 2.9209841350668955e-06,
+      "loss": 0.5101,
+      "step": 31700
+    },
+    {
+      "epoch": 0.9285484859987736,
+      "grad_norm": 4.090612888336182,
+      "learning_rate": 2.6984216944618502e-06,
+      "loss": 0.5134,
+      "step": 31800
+    },
+    {
+      "epoch": 0.9314684497912226,
+      "grad_norm": 4.454416751861572,
+      "learning_rate": 2.4845617388715823e-06,
+      "loss": 0.4935,
+      "step": 31900
+    },
+    {
+      "epoch": 0.9343884135836715,
+      "grad_norm": 2.1300718784332275,
+      "learning_rate": 2.2794233955609425e-06,
+      "loss": 0.5529,
+      "step": 32000
+    },
+    {
+      "epoch": 0.9373083773761205,
+      "grad_norm": 2.274291753768921,
+      "learning_rate": 2.0830250117488627e-06,
+      "loss": 0.5482,
+      "step": 32100
+    },
+    {
+      "epoch": 0.9402283411685695,
+      "grad_norm": 8.231354713439941,
+      "learning_rate": 1.8953841529673722e-06,
+      "loss": 0.5735,
+      "step": 32200
+    },
+    {
+      "epoch": 0.9431483049610185,
+      "grad_norm": 5.990418910980225,
+      "learning_rate": 1.7165176014906415e-06,
+      "loss": 0.5678,
+      "step": 32300
+    },
+    {
+      "epoch": 0.9460682687534675,
+      "grad_norm": 2.1669516563415527,
+      "learning_rate": 1.5464413548339162e-06,
+      "loss": 0.4608,
+      "step": 32400
+    },
+    {
+      "epoch": 0.9489882325459165,
+      "grad_norm": 4.036328315734863,
+      "learning_rate": 1.3851706243227736e-06,
+      "loss": 0.486,
+      "step": 32500
+    },
+    {
+      "epoch": 0.9519081963383654,
+      "grad_norm": 3.1790454387664795,
+      "learning_rate": 1.2327198337326762e-06,
+      "loss": 0.5589,
+      "step": 32600
+    },
+    {
+      "epoch": 0.9548281601308144,
+      "grad_norm": 4.643735408782959,
+      "learning_rate": 1.0891026179988718e-06,
+      "loss": 0.5459,
+      "step": 32700
+    },
+    {
+      "epoch": 0.9577481239232634,
+      "grad_norm": 2.757411479949951,
+      "learning_rate": 9.543318219969565e-07,
+      "loss": 0.5064,
+      "step": 32800
+    },
+    {
+      "epoch": 0.9606680877157123,
+      "grad_norm": 9.022377014160156,
+      "learning_rate": 8.284194993940065e-07,
+      "loss": 0.6064,
+      "step": 32900
+    },
+    {
+      "epoch": 0.9635880515081613,
+      "grad_norm": 4.3898210525512695,
+      "learning_rate": 7.113769115705715e-07,
+      "loss": 0.5154,
+      "step": 33000
+    },
+    {
+      "epoch": 0.9665080153006103,
+      "grad_norm": 1.2006958723068237,
+      "learning_rate": 6.032145266134493e-07,
+      "loss": 0.5003,
+      "step": 33100
+    },
+    {
+      "epoch": 0.9694279790930592,
+      "grad_norm": 3.930570602416992,
+      "learning_rate": 5.03942018379433e-07,
+      "loss": 0.5025,
+      "step": 33200
+    },
+    {
+      "epoch": 0.9723479428855082,
+      "grad_norm": 1.9354479312896729,
+      "learning_rate": 4.135682656300821e-07,
+      "loss": 0.5043,
+      "step": 33300
+    },
+    {
+      "epoch": 0.9752679066779572,
+      "grad_norm": 3.8160500526428223,
+      "learning_rate": 3.3210135123770137e-07,
+      "loss": 0.5009,
+      "step": 33400
+    },
+    {
+      "epoch": 0.9781878704704061,
+      "grad_norm": 6.7679123878479,
+      "learning_rate": 2.5954856146229723e-07,
+      "loss": 0.5898,
+      "step": 33500
+    },
+    {
+      "epoch": 0.9811078342628552,
+      "grad_norm": 2.734801769256592,
+      "learning_rate": 1.9591638530003232e-07,
+      "loss": 0.54,
+      "step": 33600
+    },
+    {
+      "epoch": 0.9840277980553042,
+      "grad_norm": 3.867943048477173,
+      "learning_rate": 1.4121051390276752e-07,
+      "loss": 0.5046,
+      "step": 33700
+    },
+    {
+      "epoch": 0.9869477618477531,
+      "grad_norm": 2.5671937465667725,
+      "learning_rate": 9.543584006909134e-08,
+      "loss": 0.609,
+      "step": 33800
+    },
+    {
+      "epoch": 0.9898677256402021,
+      "grad_norm": 3.0211405754089355,
+      "learning_rate": 5.859645780674772e-08,
+      "loss": 0.5742,
+      "step": 33900
+    },
+    {
+      "epoch": 0.992787689432651,
+      "grad_norm": 2.964229106903076,
+      "learning_rate": 3.069566196640672e-08,
+      "loss": 0.5081,
+      "step": 34000
+    },
+    {
+      "epoch": 0.9957076532251,
+      "grad_norm": 2.8453125953674316,
+      "learning_rate": 1.173594794704469e-08,
+      "loss": 0.5484,
+      "step": 34100
+    },
+    {
+      "epoch": 0.998627617017549,
+      "grad_norm": 2.5372684001922607,
+      "learning_rate": 1.7190114727116957e-09,
+      "loss": 0.4831,
+      "step": 34200
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 34247,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 0,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.5026883378073395e+17,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0251723f961cec3d43bfad005955168aa7a10cb8822875228569bb1d18b0ace
+size 5624