Training in progress, step 100, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: NousResearch/Nous-Hermes-llama-2-7b
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "NousResearch/Nous-Hermes-llama-2-7b",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "gate_proj",
+    "v_proj",
+    "up_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39b871ea4a0612201e81603c1baa4967444ab5f9fed7a543c0bb3b93ded54036
+size 1163996488

last-checkpoint/added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<pad>": 32000
+}

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c26e7c87f5ee249c1d512a8655381a3094f6241176000e7fa2b266bfcedd444c
+size 325339796

last-checkpoint/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
+size 14244

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3943847200e1e811309a0a1c7e5ee51817ba6802820b77427d9d5a2b03aa25c4
+size 1064

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,50 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,1533 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.005374754776813308,
+  "eval_steps": 500,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 5.374754776813308e-05,
+      "grad_norm": 9.018385887145996,
+      "learning_rate": 5e-09,
+      "logits/chosen": -0.5629481673240662,
+      "logits/rejected": -0.9253309965133667,
+      "logps/chosen": -80.73175048828125,
+      "logps/rejected": -120.37657165527344,
+      "loss": 1.3863,
+      "rewards/accuracies": 0.0,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0,
+      "rewards/rejected": 0.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.00010749509553626615,
+      "grad_norm": 8.409211158752441,
+      "learning_rate": 1e-08,
+      "logits/chosen": -0.6839612722396851,
+      "logits/rejected": -0.8808996677398682,
+      "logps/chosen": -89.26901245117188,
+      "logps/rejected": -102.36270904541016,
+      "loss": 1.3863,
+      "rewards/accuracies": 0.0,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0,
+      "rewards/rejected": 0.0,
+      "step": 2
+    },
+    {
+      "epoch": 0.00016124264330439922,
+      "grad_norm": 10.9932279586792,
+      "learning_rate": 1.5e-08,
+      "logits/chosen": -0.39078935980796814,
+      "logits/rejected": -1.0275822877883911,
+      "logps/chosen": -91.3505859375,
+      "logps/rejected": -113.90643310546875,
+      "loss": 1.3638,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.016756106168031693,
+      "rewards/margins": 0.023180630058050156,
+      "rewards/rejected": -0.006424522493034601,
+      "step": 3
+    },
+    {
+      "epoch": 0.0002149901910725323,
+      "grad_norm": 9.944060325622559,
+      "learning_rate": 2e-08,
+      "logits/chosen": -0.663591206073761,
+      "logits/rejected": -1.0442650318145752,
+      "logps/chosen": -68.45164489746094,
+      "logps/rejected": -113.84129333496094,
+      "loss": 1.3984,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.001784514868631959,
+      "rewards/margins": -0.011848259717226028,
+      "rewards/rejected": 0.013632774353027344,
+      "step": 4
+    },
+    {
+      "epoch": 0.0002687377388406654,
+      "grad_norm": 8.76212215423584,
+      "learning_rate": 2.5e-08,
+      "logits/chosen": -0.5241557955741882,
+      "logits/rejected": -0.9429107904434204,
+      "logps/chosen": -79.99862670898438,
+      "logps/rejected": -97.68046569824219,
+      "loss": 1.366,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.005249070934951305,
+      "rewards/margins": 0.020516324788331985,
+      "rewards/rejected": -0.015267252922058105,
+      "step": 5
+    },
+    {
+      "epoch": 0.00032248528660879845,
+      "grad_norm": 9.241307258605957,
+      "learning_rate": 3e-08,
+      "logits/chosen": -0.554192841053009,
+      "logits/rejected": -0.9173457026481628,
+      "logps/chosen": -93.0028076171875,
+      "logps/rejected": -127.36787414550781,
+      "loss": 1.379,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.0008242604089900851,
+      "rewards/margins": 0.007851887494325638,
+      "rewards/rejected": -0.007027626968920231,
+      "step": 6
+    },
+    {
+      "epoch": 0.00037623283437693153,
+      "grad_norm": 10.087677955627441,
+      "learning_rate": 3.5e-08,
+      "logits/chosen": -0.5847232341766357,
+      "logits/rejected": -0.7745344042778015,
+      "logps/chosen": -84.73774719238281,
+      "logps/rejected": -112.95906066894531,
+      "loss": 1.3738,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.0006923675537109375,
+      "rewards/margins": 0.013265704736113548,
+      "rewards/rejected": -0.013958072289824486,
+      "step": 7
+    },
+    {
+      "epoch": 0.0004299803821450646,
+      "grad_norm": 7.152434349060059,
+      "learning_rate": 4e-08,
+      "logits/chosen": -0.6588990688323975,
+      "logits/rejected": -0.7417616844177246,
+      "logps/chosen": -82.9864273071289,
+      "logps/rejected": -89.05368041992188,
+      "loss": 1.3747,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.0030961036682128906,
+      "rewards/margins": 0.011989163234829903,
+      "rewards/rejected": -0.008893061429262161,
+      "step": 8
+    },
+    {
+      "epoch": 0.0004837279299131977,
+      "grad_norm": 9.751290321350098,
+      "learning_rate": 4.5e-08,
+      "logits/chosen": -0.49971503019332886,
+      "logits/rejected": -0.7068791389465332,
+      "logps/chosen": -83.61515808105469,
+      "logps/rejected": -109.05160522460938,
+      "loss": 1.409,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.0018943313043564558,
+      "rewards/margins": -0.02229447290301323,
+      "rewards/rejected": 0.024188805371522903,
+      "step": 9
+    },
+    {
+      "epoch": 0.0005374754776813308,
+      "grad_norm": 10.492074966430664,
+      "learning_rate": 5e-08,
+      "logits/chosen": -0.5029646158218384,
+      "logits/rejected": -0.8592510223388672,
+      "logps/chosen": -96.91883850097656,
+      "logps/rejected": -131.0804443359375,
+      "loss": 1.3865,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.009151006117463112,
+      "rewards/margins": 7.531698793172836e-05,
+      "rewards/rejected": 0.009075689129531384,
+      "step": 10
+    },
+    {
+      "epoch": 0.0005912230254494639,
+      "grad_norm": 8.155610084533691,
+      "learning_rate": 5.4999999999999996e-08,
+      "logits/chosen": -0.5331442356109619,
+      "logits/rejected": -0.8224359750747681,
+      "logps/chosen": -88.8160400390625,
+      "logps/rejected": -112.42459106445312,
+      "loss": 1.3885,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.008426809683442116,
+      "rewards/margins": -0.001995563507080078,
+      "rewards/rejected": 0.010422373190522194,
+      "step": 11
+    },
+    {
+      "epoch": 0.0006449705732175969,
+      "grad_norm": 7.364086151123047,
+      "learning_rate": 6e-08,
+      "logits/chosen": -0.5735334157943726,
+      "logits/rejected": -1.0129125118255615,
+      "logps/chosen": -77.84087371826172,
+      "logps/rejected": -89.80377960205078,
+      "loss": 1.3722,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.0021553754340857267,
+      "rewards/margins": 0.014481187798082829,
+      "rewards/rejected": -0.01663656160235405,
+      "step": 12
+    },
+    {
+      "epoch": 0.00069871812098573,
+      "grad_norm": 8.48727798461914,
+      "learning_rate": 6.5e-08,
+      "logits/chosen": -0.7501606941223145,
+      "logits/rejected": -0.9889898896217346,
+      "logps/chosen": -84.73579406738281,
+      "logps/rejected": -113.06980895996094,
+      "loss": 1.3603,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.015059733763337135,
+      "rewards/margins": 0.02650745026767254,
+      "rewards/rejected": -0.011447716504335403,
+      "step": 13
+    },
+    {
+      "epoch": 0.0007524656687538631,
+      "grad_norm": 7.543310165405273,
+      "learning_rate": 7e-08,
+      "logits/chosen": -0.5986064672470093,
+      "logits/rejected": -1.0369253158569336,
+      "logps/chosen": -76.786376953125,
+      "logps/rejected": -95.89421844482422,
+      "loss": 1.3896,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.01045694388449192,
+      "rewards/margins": -0.003253889037296176,
+      "rewards/rejected": 0.013710832223296165,
+      "step": 14
+    },
+    {
+      "epoch": 0.0008062132165219961,
+      "grad_norm": 8.725497245788574,
+      "learning_rate": 7.5e-08,
+      "logits/chosen": -0.40047967433929443,
+      "logits/rejected": -0.42519611120224,
+      "logps/chosen": -91.95352172851562,
+      "logps/rejected": -109.30005645751953,
+      "loss": 1.3812,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.011234855279326439,
+      "rewards/margins": 0.005371189676225185,
+      "rewards/rejected": 0.005863667465746403,
+      "step": 15
+    },
+    {
+      "epoch": 0.0008599607642901292,
+      "grad_norm": 8.531486511230469,
+      "learning_rate": 8e-08,
+      "logits/chosen": -0.4561946988105774,
+      "logits/rejected": -0.715956449508667,
+      "logps/chosen": -75.9218521118164,
+      "logps/rejected": -120.33627319335938,
+      "loss": 1.4012,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.003715181490406394,
+      "rewards/margins": -0.014596652239561081,
+      "rewards/rejected": 0.01831183396279812,
+      "step": 16
+    },
+    {
+      "epoch": 0.0009137083120582623,
+      "grad_norm": 8.766817092895508,
+      "learning_rate": 8.500000000000001e-08,
+      "logits/chosen": -0.5665198564529419,
+      "logits/rejected": -0.5304955244064331,
+      "logps/chosen": -100.96295166015625,
+      "logps/rejected": -117.26142120361328,
+      "loss": 1.3759,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.00030202907510101795,
+      "rewards/margins": 0.010729026980698109,
+      "rewards/rejected": -0.01042699720710516,
+      "step": 17
+    },
+    {
+      "epoch": 0.0009674558598263954,
+      "grad_norm": 9.407784461975098,
+      "learning_rate": 9e-08,
+      "logits/chosen": -0.7044994831085205,
+      "logits/rejected": -0.8162285685539246,
+      "logps/chosen": -75.76687622070312,
+      "logps/rejected": -113.43864440917969,
+      "loss": 1.3908,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0002802375238388777,
+      "rewards/margins": -0.004327821545302868,
+      "rewards/rejected": 0.004047584254294634,
+      "step": 18
+    },
+    {
+      "epoch": 0.0010212034075945285,
+      "grad_norm": 7.906011581420898,
+      "learning_rate": 9.499999999999999e-08,
+      "logits/chosen": -0.6232011318206787,
+      "logits/rejected": -0.901610791683197,
+      "logps/chosen": -80.9669189453125,
+      "logps/rejected": -119.91120910644531,
+      "loss": 1.3763,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.008800078183412552,
+      "rewards/margins": 0.010257960297167301,
+      "rewards/rejected": -0.0014578821137547493,
+      "step": 19
+    },
+    {
+      "epoch": 0.0010749509553626617,
+      "grad_norm": 8.806061744689941,
+      "learning_rate": 1e-07,
+      "logits/chosen": -0.4426119029521942,
+      "logits/rejected": -1.1311769485473633,
+      "logps/chosen": -79.48087310791016,
+      "logps/rejected": -113.45155334472656,
+      "loss": 1.3945,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.0027163506019860506,
+      "rewards/margins": -0.00813446007668972,
+      "rewards/rejected": 0.005418109707534313,
+      "step": 20
+    },
+    {
+      "epoch": 0.0011286985031307946,
+      "grad_norm": 8.660501480102539,
+      "learning_rate": 1.0499999999999999e-07,
+      "logits/chosen": -0.5322842001914978,
+      "logits/rejected": -0.8683191537857056,
+      "logps/chosen": -81.68772888183594,
+      "logps/rejected": -115.72924041748047,
+      "loss": 1.3929,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.012896967120468616,
+      "rewards/margins": -0.006387948989868164,
+      "rewards/rejected": -0.006509017664939165,
+      "step": 21
+    },
+    {
+      "epoch": 0.0011824460508989278,
+      "grad_norm": 7.001013278961182,
+      "learning_rate": 1.0999999999999999e-07,
+      "logits/chosen": -0.5830490589141846,
+      "logits/rejected": -0.8426412343978882,
+      "logps/chosen": -59.00617218017578,
+      "logps/rejected": -77.81758880615234,
+      "loss": 1.3919,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0033678063191473484,
+      "rewards/margins": -0.005130481906235218,
+      "rewards/rejected": 0.00849828775972128,
+      "step": 22
+    },
+    {
+      "epoch": 0.0012361935986670608,
+      "grad_norm": 8.08579158782959,
+      "learning_rate": 1.15e-07,
+      "logits/chosen": -0.5881620645523071,
+      "logits/rejected": -0.8506753444671631,
+      "logps/chosen": -85.85636901855469,
+      "logps/rejected": -94.69215393066406,
+      "loss": 1.383,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.0047182561829686165,
+      "rewards/margins": 0.0037292949855327606,
+      "rewards/rejected": 0.0009889607317745686,
+      "step": 23
+    },
+    {
+      "epoch": 0.0012899411464351938,
+      "grad_norm": 8.288750648498535,
+      "learning_rate": 1.2e-07,
+      "logits/chosen": -0.34049859642982483,
+      "logits/rejected": -0.704309344291687,
+      "logps/chosen": -93.78474426269531,
+      "logps/rejected": -129.32717895507812,
+      "loss": 1.3858,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.0014446261338889599,
+      "rewards/margins": 0.0009137153392657638,
+      "rewards/rejected": 0.0005309106782078743,
+      "step": 24
+    },
+    {
+      "epoch": 0.001343688694203327,
+      "grad_norm": 6.940954685211182,
+      "learning_rate": 1.25e-07,
+      "logits/chosen": -0.7932472229003906,
+      "logits/rejected": -0.9097551107406616,
+      "logps/chosen": -65.62593078613281,
+      "logps/rejected": -92.70841979980469,
+      "loss": 1.3893,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.012067246250808239,
+      "rewards/margins": -0.002649521455168724,
+      "rewards/rejected": 0.014716767705976963,
+      "step": 25
+    },
+    {
+      "epoch": 0.00139743624197146,
+      "grad_norm": 9.167326927185059,
+      "learning_rate": 1.3e-07,
+      "logits/chosen": -0.5547146201133728,
+      "logits/rejected": -1.0815478563308716,
+      "logps/chosen": -100.64911651611328,
+      "logps/rejected": -103.84555053710938,
+      "loss": 1.3763,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.00014717550948262215,
+      "rewards/margins": 0.010152269154787064,
+      "rewards/rejected": -0.010005094110965729,
+      "step": 26
+    },
+    {
+      "epoch": 0.0014511837897395932,
+      "grad_norm": 8.25241470336914,
+      "learning_rate": 1.35e-07,
+      "logits/chosen": -0.5232434868812561,
+      "logits/rejected": -0.6390266418457031,
+      "logps/chosen": -91.79365539550781,
+      "logps/rejected": -116.89897155761719,
+      "loss": 1.3924,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.0075927735306322575,
+      "rewards/margins": -0.005957889836281538,
+      "rewards/rejected": -0.0016348841600120068,
+      "step": 27
+    },
+    {
+      "epoch": 0.0015049313375077261,
+      "grad_norm": 9.182291984558105,
+      "learning_rate": 1.4e-07,
+      "logits/chosen": -0.39191919565200806,
+      "logits/rejected": -0.7724546194076538,
+      "logps/chosen": -96.36701202392578,
+      "logps/rejected": -110.5954360961914,
+      "loss": 1.3972,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.008513402193784714,
+      "rewards/margins": -0.010582972317934036,
+      "rewards/rejected": 0.002069568494334817,
+      "step": 28
+    },
+    {
+      "epoch": 0.0015586788852758593,
+      "grad_norm": 9.077556610107422,
+      "learning_rate": 1.45e-07,
+      "logits/chosen": -0.5273479223251343,
+      "logits/rejected": -0.6808032989501953,
+      "logps/chosen": -98.66131591796875,
+      "logps/rejected": -151.17465209960938,
+      "loss": 1.3878,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0006317612715065479,
+      "rewards/margins": -0.0013585565611720085,
+      "rewards/rejected": 0.0019903180655092,
+      "step": 29
+    },
+    {
+      "epoch": 0.0016124264330439923,
+      "grad_norm": 7.897831916809082,
+      "learning_rate": 1.5e-07,
+      "logits/chosen": -0.7506681680679321,
+      "logits/rejected": -1.0439293384552002,
+      "logps/chosen": -70.08816528320312,
+      "logps/rejected": -110.37745666503906,
+      "loss": 1.3898,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.0013779643923044205,
+      "rewards/margins": -0.003206300549209118,
+      "rewards/rejected": 0.004584264941513538,
+      "step": 30
+    },
+    {
+      "epoch": 0.0016661739808121255,
+      "grad_norm": 8.56389045715332,
+      "learning_rate": 1.55e-07,
+      "logits/chosen": -0.5107775330543518,
+      "logits/rejected": -0.7187220454216003,
+      "logps/chosen": -106.04706573486328,
+      "logps/rejected": -110.80841064453125,
+      "loss": 1.3777,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.006948089227080345,
+      "rewards/margins": 0.008790111169219017,
+      "rewards/rejected": -0.0018420221749693155,
+      "step": 31
+    },
+    {
+      "epoch": 0.0017199215285802585,
+      "grad_norm": 7.16372013092041,
+      "learning_rate": 1.6e-07,
+      "logits/chosen": -0.6406233906745911,
+      "logits/rejected": -0.7985771894454956,
+      "logps/chosen": -79.37033081054688,
+      "logps/rejected": -100.94114685058594,
+      "loss": 1.4107,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.006321430206298828,
+      "rewards/margins": -0.024100303649902344,
+      "rewards/rejected": 0.017778873443603516,
+      "step": 32
+    },
+    {
+      "epoch": 0.0017736690763483917,
+      "grad_norm": 7.901755332946777,
+      "learning_rate": 1.65e-07,
+      "logits/chosen": -0.777077317237854,
+      "logits/rejected": -0.8622071743011475,
+      "logps/chosen": -67.00970458984375,
+      "logps/rejected": -92.73141479492188,
+      "loss": 1.3792,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0007941480726003647,
+      "rewards/margins": 0.007220578379929066,
+      "rewards/rejected": -0.006426429376006126,
+      "step": 33
+    },
+    {
+      "epoch": 0.0018274166241165246,
+      "grad_norm": 7.719196796417236,
+      "learning_rate": 1.7000000000000001e-07,
+      "logits/chosen": -0.2993572950363159,
+      "logits/rejected": -0.6201244592666626,
+      "logps/chosen": -85.47561645507812,
+      "logps/rejected": -100.66415405273438,
+      "loss": 1.3969,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.003897094866260886,
+      "rewards/margins": -0.01031260471791029,
+      "rewards/rejected": 0.0064155105501413345,
+      "step": 34
+    },
+    {
+      "epoch": 0.0018811641718846578,
+      "grad_norm": 11.600595474243164,
+      "learning_rate": 1.75e-07,
+      "logits/chosen": -0.6471947431564331,
+      "logits/rejected": -0.9437248706817627,
+      "logps/chosen": -112.87066650390625,
+      "logps/rejected": -130.8255615234375,
+      "loss": 1.3927,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.015840481966733932,
+      "rewards/margins": -0.005790282040834427,
+      "rewards/rejected": -0.010050201788544655,
+      "step": 35
+    },
+    {
+      "epoch": 0.0019349117196527908,
+      "grad_norm": 8.569315910339355,
+      "learning_rate": 1.8e-07,
+      "logits/chosen": -0.5607515573501587,
+      "logits/rejected": -0.9741392731666565,
+      "logps/chosen": -89.04241180419922,
+      "logps/rejected": -141.5797119140625,
+      "loss": 1.3852,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.009252786636352539,
+      "rewards/margins": 0.0013018138706684113,
+      "rewards/rejected": 0.007950973697006702,
+      "step": 36
+    },
+    {
+      "epoch": 0.0019886592674209238,
+      "grad_norm": 7.82846736907959,
+      "learning_rate": 1.85e-07,
+      "logits/chosen": -0.3905686140060425,
+      "logits/rejected": -1.1150908470153809,
+      "logps/chosen": -89.02799987792969,
+      "logps/rejected": -101.53260803222656,
+      "loss": 1.3894,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.007035780698060989,
+      "rewards/margins": -0.002919721882790327,
+      "rewards/rejected": -0.004116058815270662,
+      "step": 37
+    },
+    {
+      "epoch": 0.002042406815189057,
+      "grad_norm": 8.96768569946289,
+      "learning_rate": 1.8999999999999998e-07,
+      "logits/chosen": -0.4186624586582184,
+      "logits/rejected": -0.7087715864181519,
+      "logps/chosen": -78.67716217041016,
+      "logps/rejected": -105.57879638671875,
+      "loss": 1.3895,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": 0.0027568337973207235,
+      "rewards/margins": -0.003093290375545621,
+      "rewards/rejected": 0.0058501241728663445,
+      "step": 38
+    },
+    {
+      "epoch": 0.00209615436295719,
+      "grad_norm": 7.81805419921875,
+      "learning_rate": 1.9499999999999999e-07,
+      "logits/chosen": -0.7153301239013672,
+      "logits/rejected": -0.6971107721328735,
+      "logps/chosen": -72.32463836669922,
+      "logps/rejected": -105.63674926757812,
+      "loss": 1.3729,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.009188270196318626,
+      "rewards/margins": 0.01365575846284628,
+      "rewards/rejected": -0.004467487800866365,
+      "step": 39
+    },
+    {
+      "epoch": 0.0021499019107253233,
+      "grad_norm": 8.586557388305664,
+      "learning_rate": 2e-07,
+      "logits/chosen": -0.5092922449111938,
+      "logits/rejected": -0.8808071613311768,
+      "logps/chosen": -81.26252746582031,
+      "logps/rejected": -105.72013092041016,
+      "loss": 1.3932,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.01609921269118786,
+      "rewards/margins": -0.006821251008659601,
+      "rewards/rejected": -0.009277964010834694,
+      "step": 40
+    },
+    {
+      "epoch": 0.002203649458493456,
+      "grad_norm": 7.373818397521973,
+      "learning_rate": 2.0499999999999997e-07,
+      "logits/chosen": -0.49268585443496704,
+      "logits/rejected": -0.7908859252929688,
+      "logps/chosen": -86.733642578125,
+      "logps/rejected": -95.32139587402344,
+      "loss": 1.369,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.009233569726347923,
+      "rewards/margins": 0.017481137067079544,
+      "rewards/rejected": -0.00824756734073162,
+      "step": 41
+    },
+    {
+      "epoch": 0.0022573970062615893,
+      "grad_norm": 8.270150184631348,
+      "learning_rate": 2.0999999999999997e-07,
+      "logits/chosen": -0.5285747051239014,
+      "logits/rejected": -0.722440242767334,
+      "logps/chosen": -83.8260726928711,
+      "logps/rejected": -125.6030502319336,
+      "loss": 1.385,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.007476449478417635,
+      "rewards/margins": 0.0014731173869222403,
+      "rewards/rejected": 0.006003332324326038,
+      "step": 42
+    },
+    {
+      "epoch": 0.0023111445540297225,
+      "grad_norm": 10.629551887512207,
+      "learning_rate": 2.1499999999999998e-07,
+      "logits/chosen": -0.49013882875442505,
+      "logits/rejected": -0.6852960586547852,
+      "logps/chosen": -105.13156127929688,
+      "logps/rejected": -123.69252014160156,
+      "loss": 1.3746,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.009237194433808327,
+      "rewards/margins": 0.012375927530229092,
+      "rewards/rejected": -0.0031387328635901213,
+      "step": 43
+    },
+    {
+      "epoch": 0.0023648921017978557,
+      "grad_norm": 7.858402252197266,
+      "learning_rate": 2.1999999999999998e-07,
+      "logits/chosen": -0.5702916383743286,
+      "logits/rejected": -0.8965415954589844,
+      "logps/chosen": -68.97383117675781,
+      "logps/rejected": -101.99735260009766,
+      "loss": 1.3733,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.005028081592172384,
+      "rewards/margins": 0.013224005699157715,
+      "rewards/rejected": -0.008195924572646618,
+      "step": 44
+    },
+    {
+      "epoch": 0.0024186396495659884,
+      "grad_norm": 8.891136169433594,
+      "learning_rate": 2.25e-07,
+      "logits/chosen": -0.6039612293243408,
+      "logits/rejected": -1.0555522441864014,
+      "logps/chosen": -93.02165222167969,
+      "logps/rejected": -129.453857421875,
+      "loss": 1.3752,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.009529518894851208,
+      "rewards/margins": 0.011351561173796654,
+      "rewards/rejected": -0.001822042977437377,
+      "step": 45
+    },
+    {
+      "epoch": 0.0024723871973341216,
+      "grad_norm": 9.362869262695312,
+      "learning_rate": 2.3e-07,
+      "logits/chosen": -0.7231209874153137,
+      "logits/rejected": -0.6981338858604431,
+      "logps/chosen": -94.05708312988281,
+      "logps/rejected": -139.45484924316406,
+      "loss": 1.3876,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.02146873250603676,
+      "rewards/margins": -0.0009884836617857218,
+      "rewards/rejected": 0.022457217797636986,
+      "step": 46
+    },
+    {
+      "epoch": 0.002526134745102255,
+      "grad_norm": 8.48691177368164,
+      "learning_rate": 2.3499999999999997e-07,
+      "logits/chosen": -0.7336191534996033,
+      "logits/rejected": -0.9868717193603516,
+      "logps/chosen": -75.6761474609375,
+      "logps/rejected": -105.52317810058594,
+      "loss": 1.3536,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.00992603413760662,
+      "rewards/margins": 0.03316822275519371,
+      "rewards/rejected": -0.02324218861758709,
+      "step": 47
+    },
+    {
+      "epoch": 0.0025798822928703876,
+      "grad_norm": 7.733593940734863,
+      "learning_rate": 2.4e-07,
+      "logits/chosen": -0.5540213584899902,
+      "logits/rejected": -0.8223700523376465,
+      "logps/chosen": -71.37220001220703,
+      "logps/rejected": -81.74217224121094,
+      "loss": 1.4,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.0020942208357155323,
+      "rewards/margins": -0.013409614562988281,
+      "rewards/rejected": 0.011315394192934036,
+      "step": 48
+    },
+    {
+      "epoch": 0.0026336298406385208,
+      "grad_norm": 9.125642776489258,
+      "learning_rate": 2.45e-07,
+      "logits/chosen": -0.49975132942199707,
+      "logits/rejected": -0.7237377166748047,
+      "logps/chosen": -79.24974822998047,
+      "logps/rejected": -149.38690185546875,
+      "loss": 1.3972,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.013892841525375843,
+      "rewards/margins": -0.010572625324130058,
+      "rewards/rejected": -0.0033202162012457848,
+      "step": 49
+    },
+    {
+      "epoch": 0.002687377388406654,
+      "grad_norm": 7.797073841094971,
+      "learning_rate": 2.5e-07,
+      "logits/chosen": -0.7408368587493896,
+      "logits/rejected": -0.6117101907730103,
+      "logps/chosen": -96.25028228759766,
+      "logps/rejected": -127.87377166748047,
+      "loss": 1.3682,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.006863451562821865,
+      "rewards/margins": 0.018326900899410248,
+      "rewards/rejected": -0.011463451199233532,
+      "step": 50
+    },
+    {
+      "epoch": 0.002741124936174787,
+      "grad_norm": 8.364821434020996,
+      "learning_rate": 2.55e-07,
+      "logits/chosen": -0.5893720388412476,
+      "logits/rejected": -0.9179937243461609,
+      "logps/chosen": -94.17994689941406,
+      "logps/rejected": -111.77970123291016,
+      "loss": 1.4037,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.005445909686386585,
+      "rewards/margins": -0.016887545585632324,
+      "rewards/rejected": 0.011441635899245739,
+      "step": 51
+    },
+    {
+      "epoch": 0.00279487248394292,
+      "grad_norm": 8.976974487304688,
+      "learning_rate": 2.6e-07,
+      "logits/chosen": -0.5119687914848328,
+      "logits/rejected": -0.8165309429168701,
+      "logps/chosen": -82.34027099609375,
+      "logps/rejected": -134.41204833984375,
+      "loss": 1.3866,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.0027532577514648438,
+      "rewards/margins": -0.00021648360416293144,
+      "rewards/rejected": 0.002969742054119706,
+      "step": 52
+    },
+    {
+      "epoch": 0.002848620031711053,
+      "grad_norm": 9.535009384155273,
+      "learning_rate": 2.65e-07,
+      "logits/chosen": -0.48858749866485596,
+      "logits/rejected": -0.7641533613204956,
+      "logps/chosen": -108.45321655273438,
+      "logps/rejected": -134.2806854248047,
+      "loss": 1.388,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.0011934274807572365,
+      "rewards/margins": -0.0014282232150435448,
+      "rewards/rejected": 0.00023479480296373367,
+      "step": 53
+    },
+    {
+      "epoch": 0.0029023675794791863,
+      "grad_norm": 7.457547187805176,
+      "learning_rate": 2.7e-07,
+      "logits/chosen": -0.6441543698310852,
+      "logits/rejected": -0.5924786329269409,
+      "logps/chosen": -88.5523910522461,
+      "logps/rejected": -101.81995391845703,
+      "loss": 1.3794,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0008887768490239978,
+      "rewards/margins": 0.007016850169748068,
+      "rewards/rejected": -0.0079056266695261,
+      "step": 54
+    },
+    {
+      "epoch": 0.0029561151272473195,
+      "grad_norm": 9.02698040008545,
+      "learning_rate": 2.75e-07,
+      "logits/chosen": -0.468053936958313,
+      "logits/rejected": -0.9804271459579468,
+      "logps/chosen": -94.00070190429688,
+      "logps/rejected": -149.0098419189453,
+      "loss": 1.4167,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.015568828210234642,
+      "rewards/margins": -0.029503628611564636,
+      "rewards/rejected": 0.013934802263975143,
+      "step": 55
+    },
+    {
+      "epoch": 0.0030098626750154523,
+      "grad_norm": 10.612890243530273,
+      "learning_rate": 2.8e-07,
+      "logits/chosen": -0.6267477869987488,
+      "logits/rejected": -0.8941481113433838,
+      "logps/chosen": -84.3619155883789,
+      "logps/rejected": -106.56355285644531,
+      "loss": 1.3721,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.010368634015321732,
+      "rewards/margins": 0.014438199810683727,
+      "rewards/rejected": -0.004069566261023283,
+      "step": 56
+    },
+    {
+      "epoch": 0.0030636102227835854,
+      "grad_norm": 9.122997283935547,
+      "learning_rate": 2.8499999999999997e-07,
+      "logits/chosen": -0.4833652377128601,
+      "logits/rejected": -0.5830209255218506,
+      "logps/chosen": -96.44157409667969,
+      "logps/rejected": -137.86610412597656,
+      "loss": 1.3913,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.00467534177005291,
+      "rewards/margins": -0.004872847348451614,
+      "rewards/rejected": 0.0001975062768906355,
+      "step": 57
+    },
+    {
+      "epoch": 0.0031173577705517186,
+      "grad_norm": 7.18145227432251,
+      "learning_rate": 2.9e-07,
+      "logits/chosen": -0.5169291496276855,
+      "logits/rejected": -0.7388797402381897,
+      "logps/chosen": -77.57707214355469,
+      "logps/rejected": -91.94204711914062,
+      "loss": 1.3668,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.019289113581180573,
+      "rewards/margins": 0.019966794177889824,
+      "rewards/rejected": -0.0006776812952011824,
+      "step": 58
+    },
+    {
+      "epoch": 0.003171105318319852,
+      "grad_norm": 8.925081253051758,
+      "learning_rate": 2.95e-07,
+      "logits/chosen": -0.6823402643203735,
+      "logits/rejected": -0.8340035676956177,
+      "logps/chosen": -93.4341812133789,
+      "logps/rejected": -125.25653076171875,
+      "loss": 1.402,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.018242502585053444,
+      "rewards/margins": -0.015286922454833984,
+      "rewards/rejected": -0.0029555796645581722,
+      "step": 59
+    },
+    {
+      "epoch": 0.0032248528660879846,
+      "grad_norm": 8.46120834350586,
+      "learning_rate": 3e-07,
+      "logits/chosen": -0.5364607572555542,
+      "logits/rejected": -0.74739670753479,
+      "logps/chosen": -73.58871459960938,
+      "logps/rejected": -86.86251831054688,
+      "loss": 1.4047,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.010658741928637028,
+      "rewards/margins": -0.018239475786685944,
+      "rewards/rejected": 0.007580732926726341,
+      "step": 60
+    },
+    {
+      "epoch": 0.0032786004138561178,
+      "grad_norm": 11.018837928771973,
+      "learning_rate": 3.05e-07,
+      "logits/chosen": -0.529459536075592,
+      "logits/rejected": -1.0881057977676392,
+      "logps/chosen": -92.7784652709961,
+      "logps/rejected": -140.6796875,
+      "loss": 1.3855,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.007517814636230469,
+      "rewards/margins": 0.0014034267514944077,
+      "rewards/rejected": -0.008921242319047451,
+      "step": 61
+    },
+    {
+      "epoch": 0.003332347961624251,
+      "grad_norm": 7.908779621124268,
+      "learning_rate": 3.1e-07,
+      "logits/chosen": -0.3675628900527954,
+      "logits/rejected": -0.7041562795639038,
+      "logps/chosen": -72.2803955078125,
+      "logps/rejected": -98.57533264160156,
+      "loss": 1.3855,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0011669636005535722,
+      "rewards/margins": 0.0009401319548487663,
+      "rewards/rejected": -0.0021070956718176603,
+      "step": 62
+    },
+    {
+      "epoch": 0.003386095509392384,
+      "grad_norm": 10.110648155212402,
+      "learning_rate": 3.15e-07,
+      "logits/chosen": -0.7167191505432129,
+      "logits/rejected": -0.7103843688964844,
+      "logps/chosen": -72.10398864746094,
+      "logps/rejected": -112.51895904541016,
+      "loss": 1.3809,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.004012250807136297,
+      "rewards/margins": 0.005618619732558727,
+      "rewards/rejected": -0.0016063684597611427,
+      "step": 63
+    },
+    {
+      "epoch": 0.003439843057160517,
+      "grad_norm": 9.034467697143555,
+      "learning_rate": 3.2e-07,
+      "logits/chosen": -0.7524758577346802,
+      "logits/rejected": -0.7390173673629761,
+      "logps/chosen": -71.0537109375,
+      "logps/rejected": -110.75092315673828,
+      "loss": 1.3879,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.003200674429535866,
+      "rewards/margins": -0.0012365346774458885,
+      "rewards/rejected": -0.0019641402177512646,
+      "step": 64
+    },
+    {
+      "epoch": 0.00349359060492865,
+      "grad_norm": 6.976632118225098,
+      "learning_rate": 3.25e-07,
+      "logits/chosen": -0.8358469009399414,
+      "logits/rejected": -1.016180157661438,
+      "logps/chosen": -66.08577728271484,
+      "logps/rejected": -89.9134750366211,
+      "loss": 1.3842,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.010739064775407314,
+      "rewards/margins": 0.0022439719177782536,
+      "rewards/rejected": 0.008495092391967773,
+      "step": 65
+    },
+    {
+      "epoch": 0.0035473381526967833,
+      "grad_norm": 8.591444969177246,
+      "learning_rate": 3.3e-07,
+      "logits/chosen": -0.6935573220252991,
+      "logits/rejected": -0.6134279370307922,
+      "logps/chosen": -88.94845581054688,
+      "logps/rejected": -104.41876220703125,
+      "loss": 1.3835,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0005981442518532276,
+      "rewards/margins": 0.0029290677048265934,
+      "rewards/rejected": -0.002330922521650791,
+      "step": 66
+    },
+    {
+      "epoch": 0.0036010857004649165,
+      "grad_norm": 7.711324214935303,
+      "learning_rate": 3.35e-07,
+      "logits/chosen": -0.7065850496292114,
+      "logits/rejected": -0.5953470468521118,
+      "logps/chosen": -89.66769409179688,
+      "logps/rejected": -95.72993469238281,
+      "loss": 1.3829,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.0043515912257134914,
+      "rewards/margins": 0.0034556628670543432,
+      "rewards/rejected": 0.0008959295228123665,
+      "step": 67
+    },
+    {
+      "epoch": 0.0036548332482330493,
+      "grad_norm": 9.339879035949707,
+      "learning_rate": 3.4000000000000003e-07,
+      "logits/chosen": -0.5466330051422119,
+      "logits/rejected": -0.9444475173950195,
+      "logps/chosen": -88.27287292480469,
+      "logps/rejected": -127.44645690917969,
+      "loss": 1.387,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0009190088603645563,
+      "rewards/margins": -0.0006193636218085885,
+      "rewards/rejected": -0.0002996448893100023,
+      "step": 68
+    },
+    {
+      "epoch": 0.0037085807960011825,
+      "grad_norm": 9.99643611907959,
+      "learning_rate": 3.45e-07,
+      "logits/chosen": -0.633792519569397,
+      "logits/rejected": -0.6285285949707031,
+      "logps/chosen": -102.36030578613281,
+      "logps/rejected": -129.0988311767578,
+      "loss": 1.3821,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.0004914281889796257,
+      "rewards/margins": 0.0044272420927882195,
+      "rewards/rejected": -0.0049186707474291325,
+      "step": 69
+    },
+    {
+      "epoch": 0.0037623283437693156,
+      "grad_norm": 9.309488296508789,
+      "learning_rate": 3.5e-07,
+      "logits/chosen": -0.3174594044685364,
+      "logits/rejected": -0.789804995059967,
+      "logps/chosen": -84.69456481933594,
+      "logps/rejected": -107.91668701171875,
+      "loss": 1.3789,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.012507820501923561,
+      "rewards/margins": 0.007670545484870672,
+      "rewards/rejected": 0.0048372745513916016,
+      "step": 70
+    },
+    {
+      "epoch": 0.0038160758915374484,
+      "grad_norm": 8.66744613647461,
+      "learning_rate": 3.55e-07,
+      "logits/chosen": -0.8024520874023438,
+      "logits/rejected": -0.7307050228118896,
+      "logps/chosen": -76.72541809082031,
+      "logps/rejected": -106.57899475097656,
+      "loss": 1.3894,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.012390756979584694,
+      "rewards/margins": -0.0021251197904348373,
+      "rewards/rejected": 0.014515876770019531,
+      "step": 71
+    },
+    {
+      "epoch": 0.0038698234393055816,
+      "grad_norm": 9.889826774597168,
+      "learning_rate": 3.6e-07,
+      "logits/chosen": -0.5179650783538818,
+      "logits/rejected": -0.8690033555030823,
+      "logps/chosen": -81.47711181640625,
+      "logps/rejected": -150.914306640625,
+      "loss": 1.3664,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.004861020483076572,
+      "rewards/margins": 0.020183132961392403,
+      "rewards/rejected": -0.015322113409638405,
+      "step": 72
+    },
+    {
+      "epoch": 0.003923570987073715,
+      "grad_norm": 8.243026733398438,
+      "learning_rate": 3.65e-07,
+      "logits/chosen": -0.3261711597442627,
+      "logits/rejected": -0.6148155331611633,
+      "logps/chosen": -84.84300231933594,
+      "logps/rejected": -118.35273742675781,
+      "loss": 1.3865,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.009526778012514114,
+      "rewards/margins": 0.00015711761079728603,
+      "rewards/rejected": 0.00936965923756361,
+      "step": 73
+    },
+    {
+      "epoch": 0.0039773185348418475,
+      "grad_norm": 9.10533332824707,
+      "learning_rate": 3.7e-07,
+      "logits/chosen": -0.5265448689460754,
+      "logits/rejected": -0.4441685676574707,
+      "logps/chosen": -101.1104736328125,
+      "logps/rejected": -118.29222106933594,
+      "loss": 1.3782,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.01149735413491726,
+      "rewards/margins": 0.008614778518676758,
+      "rewards/rejected": 0.002882576547563076,
+      "step": 74
+    },
+    {
+      "epoch": 0.004031066082609981,
+      "grad_norm": 8.713000297546387,
+      "learning_rate": 3.75e-07,
+      "logits/chosen": -0.8108004331588745,
+      "logits/rejected": -0.740679144859314,
+      "logps/chosen": -93.86286926269531,
+      "logps/rejected": -128.41632080078125,
+      "loss": 1.3941,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.0016858575399965048,
+      "rewards/margins": -0.007669306360185146,
+      "rewards/rejected": 0.005983447656035423,
+      "step": 75
+    },
+    {
+      "epoch": 0.004084813630378114,
+      "grad_norm": 9.181607246398926,
+      "learning_rate": 3.7999999999999996e-07,
+      "logits/chosen": -0.47283488512039185,
+      "logits/rejected": -0.935647189617157,
+      "logps/chosen": -95.63761901855469,
+      "logps/rejected": -115.92280578613281,
+      "loss": 1.3933,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.0005431172903627157,
+      "rewards/margins": -0.006902456283569336,
+      "rewards/rejected": 0.006359338294714689,
+      "step": 76
+    },
+    {
+      "epoch": 0.004138561178146247,
+      "grad_norm": 6.962100982666016,
+      "learning_rate": 3.8499999999999997e-07,
+      "logits/chosen": -0.6948473453521729,
+      "logits/rejected": -1.3128026723861694,
+      "logps/chosen": -71.61614227294922,
+      "logps/rejected": -90.48023223876953,
+      "loss": 1.3758,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.00525279063731432,
+      "rewards/margins": 0.010612440295517445,
+      "rewards/rejected": -0.005359649192541838,
+      "step": 77
+    },
+    {
+      "epoch": 0.00419230872591438,
+      "grad_norm": 8.91589069366455,
+      "learning_rate": 3.8999999999999997e-07,
+      "logits/chosen": -0.3106628954410553,
+      "logits/rejected": -0.6929190158843994,
+      "logps/chosen": -80.65679168701172,
+      "logps/rejected": -115.49663543701172,
+      "loss": 1.3935,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.009992456063628197,
+      "rewards/margins": -0.006991195492446423,
+      "rewards/rejected": -0.003001260804012418,
+      "step": 78
+    },
+    {
+      "epoch": 0.004246056273682513,
+      "grad_norm": 9.223258018493652,
+      "learning_rate": 3.95e-07,
+      "logits/chosen": -0.5829489827156067,
+      "logits/rejected": -0.9253657460212708,
+      "logps/chosen": -97.073974609375,
+      "logps/rejected": -119.96102142333984,
+      "loss": 1.3855,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.0019248956814408302,
+      "rewards/margins": 0.0013014795258641243,
+      "rewards/rejected": 0.0006234167376533151,
+      "step": 79
+    },
+    {
+      "epoch": 0.004299803821450647,
+      "grad_norm": 7.335805892944336,
+      "learning_rate": 4e-07,
+      "logits/chosen": -0.6180788278579712,
+      "logits/rejected": -0.8246430158615112,
+      "logps/chosen": -65.96183776855469,
+      "logps/rejected": -90.71397399902344,
+      "loss": 1.3685,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.011597157455980778,
+      "rewards/margins": 0.018112804740667343,
+      "rewards/rejected": -0.006515646353363991,
+      "step": 80
+    },
+    {
+      "epoch": 0.0043535513692187795,
+      "grad_norm": 9.195491790771484,
+      "learning_rate": 4.05e-07,
+      "logits/chosen": -0.815979540348053,
+      "logits/rejected": -0.7271102070808411,
+      "logps/chosen": -91.46857452392578,
+      "logps/rejected": -138.6331329345703,
+      "loss": 1.3745,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.007442760281264782,
+      "rewards/margins": 0.012495232746005058,
+      "rewards/rejected": -0.005052470602095127,
+      "step": 81
+    },
+    {
+      "epoch": 0.004407298916986912,
+      "grad_norm": 8.091022491455078,
+      "learning_rate": 4.0999999999999994e-07,
+      "logits/chosen": -0.49595868587493896,
+      "logits/rejected": -0.903720498085022,
+      "logps/chosen": -87.2410659790039,
+      "logps/rejected": -123.95703125,
+      "loss": 1.3733,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.0067462921142578125,
+      "rewards/margins": 0.013173293322324753,
+      "rewards/rejected": -0.019919587299227715,
+      "step": 82
+    },
+    {
+      "epoch": 0.004461046464755046,
+      "grad_norm": 8.640301704406738,
+      "learning_rate": 4.1499999999999994e-07,
+      "logits/chosen": -0.7808905839920044,
+      "logits/rejected": -0.9601308107376099,
+      "logps/chosen": -92.71270751953125,
+      "logps/rejected": -103.0289306640625,
+      "loss": 1.392,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.00455665634945035,
+      "rewards/margins": -0.005107973702251911,
+      "rewards/rejected": 0.0005513187497854233,
+      "step": 83
+    },
+    {
+      "epoch": 0.004514794012523179,
+      "grad_norm": 10.411420822143555,
+      "learning_rate": 4.1999999999999995e-07,
+      "logits/chosen": -0.5638555288314819,
+      "logits/rejected": -0.7834076285362244,
+      "logps/chosen": -104.28919982910156,
+      "logps/rejected": -127.33184814453125,
+      "loss": 1.3779,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.003203297033905983,
+      "rewards/margins": 0.008511114865541458,
+      "rewards/rejected": -0.011714410968124866,
+      "step": 84
+    },
+    {
+      "epoch": 0.004568541560291311,
+      "grad_norm": 8.805337905883789,
+      "learning_rate": 4.2499999999999995e-07,
+      "logits/chosen": -0.5656499862670898,
+      "logits/rejected": -0.790486216545105,
+      "logps/chosen": -89.9202651977539,
+      "logps/rejected": -131.35479736328125,
+      "loss": 1.3817,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.00927596166729927,
+      "rewards/margins": 0.004767702892422676,
+      "rewards/rejected": -0.014043664559721947,
+      "step": 85
+    },
+    {
+      "epoch": 0.004622289108059445,
+      "grad_norm": 8.494221687316895,
+      "learning_rate": 4.2999999999999996e-07,
+      "logits/chosen": -0.7255963087081909,
+      "logits/rejected": -1.2127760648727417,
+      "logps/chosen": -81.29212951660156,
+      "logps/rejected": -100.43565368652344,
+      "loss": 1.398,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.0021460053976625204,
+      "rewards/margins": -0.010950803756713867,
+      "rewards/rejected": 0.013096809387207031,
+      "step": 86
+    },
+    {
+      "epoch": 0.004676036655827578,
+      "grad_norm": 9.525997161865234,
+      "learning_rate": 4.3499999999999996e-07,
+      "logits/chosen": -0.67186439037323,
+      "logits/rejected": -0.680492103099823,
+      "logps/chosen": -83.36714172363281,
+      "logps/rejected": -120.49726104736328,
+      "loss": 1.3834,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": 0.008803272619843483,
+      "rewards/margins": 0.003020667703822255,
+      "rewards/rejected": 0.005782604217529297,
+      "step": 87
+    },
+    {
+      "epoch": 0.004729784203595711,
+      "grad_norm": 10.27984619140625,
+      "learning_rate": 4.3999999999999997e-07,
+      "logits/chosen": -0.6513060331344604,
+      "logits/rejected": -0.6883199214935303,
+      "logps/chosen": -75.36994934082031,
+      "logps/rejected": -114.74723815917969,
+      "loss": 1.3685,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.0018049958162009716,
+      "rewards/margins": 0.018433403223752975,
+      "rewards/rejected": -0.016628408804535866,
+      "step": 88
+    },
+    {
+      "epoch": 0.004783531751363844,
+      "grad_norm": 8.953598976135254,
+      "learning_rate": 4.45e-07,
+      "logits/chosen": -0.7916650772094727,
+      "logits/rejected": -1.2761249542236328,
+      "logps/chosen": -98.39886474609375,
+      "logps/rejected": -123.22477722167969,
+      "loss": 1.3888,
+      "rewards/accuracies": 0.25,
+      "rewards/chosen": -0.013754701241850853,
+      "rewards/margins": -0.0023045053239911795,
+      "rewards/rejected": -0.01145019568502903,
+      "step": 89
+    },
+    {
+      "epoch": 0.004837279299131977,
+      "grad_norm": 9.589315414428711,
+      "learning_rate": 4.5e-07,
+      "logits/chosen": -0.6036202311515808,
+      "logits/rejected": -0.9316726922988892,
+      "logps/chosen": -93.84979248046875,
+      "logps/rejected": -125.3914794921875,
+      "loss": 1.3902,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.016410110518336296,
+      "rewards/margins": -0.0034947870299220085,
+      "rewards/rejected": -0.012915325351059437,
+      "step": 90
+    },
+    {
+      "epoch": 0.0048910268469001105,
+      "grad_norm": 8.582666397094727,
+      "learning_rate": 4.55e-07,
+      "logits/chosen": -0.9905259609222412,
+      "logits/rejected": -1.2266937494277954,
+      "logps/chosen": -79.10990905761719,
+      "logps/rejected": -105.07563781738281,
+      "loss": 1.3924,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": 0.002523183822631836,
+      "rewards/margins": -0.005945872515439987,
+      "rewards/rejected": 0.008469056338071823,
+      "step": 91
+    },
+    {
+      "epoch": 0.004944774394668243,
+      "grad_norm": 8.097585678100586,
+      "learning_rate": 4.6e-07,
+      "logits/chosen": -0.7338860034942627,
+      "logits/rejected": -0.5818134546279907,
+      "logps/chosen": -87.86322021484375,
+      "logps/rejected": -100.40827941894531,
+      "loss": 1.3985,
+      "rewards/accuracies": 0.375,
+      "rewards/chosen": -0.002400016877800226,
+      "rewards/margins": -0.011916255578398705,
+      "rewards/rejected": 0.009516239166259766,
+      "step": 92
+    },
+    {
+      "epoch": 0.004998521942436376,
+      "grad_norm": 8.177687644958496,
+      "learning_rate": 4.65e-07,
+      "logits/chosen": -0.5757491588592529,
+      "logits/rejected": -0.8946194648742676,
+      "logps/chosen": -70.17253112792969,
+      "logps/rejected": -105.48762512207031,
+      "loss": 1.3723,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.020789574831724167,
+      "rewards/margins": 0.014466093853116035,
+      "rewards/rejected": 0.006323480512946844,
+      "step": 93
+    },
+    {
+      "epoch": 0.00505226949020451,
+      "grad_norm": 8.56423282623291,
+      "learning_rate": 4.6999999999999995e-07,
+      "logits/chosen": -0.5390354990959167,
+      "logits/rejected": -0.40581464767456055,
+      "logps/chosen": -82.2030029296875,
+      "logps/rejected": -110.38099670410156,
+      "loss": 1.3495,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.020420502871274948,
+      "rewards/margins": 0.03794069588184357,
+      "rewards/rejected": -0.01752018928527832,
+      "step": 94
+    },
+    {
+      "epoch": 0.005106017037972642,
+      "grad_norm": 8.461507797241211,
+      "learning_rate": 4.7499999999999995e-07,
+      "logits/chosen": -0.6540727615356445,
+      "logits/rejected": -0.9722909927368164,
+      "logps/chosen": -103.54087829589844,
+      "logps/rejected": -112.89604187011719,
+      "loss": 1.3667,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.0024640089832246304,
+      "rewards/margins": 0.01972184330224991,
+      "rewards/rejected": -0.01725783385336399,
+      "step": 95
+    },
+    {
+      "epoch": 0.005159764585740775,
+      "grad_norm": 8.544709205627441,
+      "learning_rate": 4.8e-07,
+      "logits/chosen": -1.1151082515716553,
+      "logits/rejected": -0.8405168652534485,
+      "logps/chosen": -93.03946685791016,
+      "logps/rejected": -121.34730529785156,
+      "loss": 1.3508,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.015212059952318668,
+      "rewards/margins": 0.03674693405628204,
+      "rewards/rejected": -0.02153487130999565,
+      "step": 96
+    },
+    {
+      "epoch": 0.005213512133508909,
+      "grad_norm": 11.145546913146973,
+      "learning_rate": 4.85e-07,
+      "logits/chosen": -0.8847471475601196,
+      "logits/rejected": -1.0487537384033203,
+      "logps/chosen": -101.20974731445312,
+      "logps/rejected": -154.65463256835938,
+      "loss": 1.3733,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": 0.01565098948776722,
+      "rewards/margins": 0.013369608670473099,
+      "rewards/rejected": 0.002281379420310259,
+      "step": 97
+    },
+    {
+      "epoch": 0.0052672596812770416,
+      "grad_norm": 8.07213306427002,
+      "learning_rate": 4.9e-07,
+      "logits/chosen": -0.6318795084953308,
+      "logits/rejected": -0.9646536111831665,
+      "logps/chosen": -95.36701965332031,
+      "logps/rejected": -107.36691284179688,
+      "loss": 1.3728,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.012050819583237171,
+      "rewards/margins": 0.013714075088500977,
+      "rewards/rejected": -0.001663255738094449,
+      "step": 98
+    },
+    {
+      "epoch": 0.005321007229045175,
+      "grad_norm": 9.691338539123535,
+      "learning_rate": 4.95e-07,
+      "logits/chosen": -0.4308117628097534,
+      "logits/rejected": -0.869768500328064,
+      "logps/chosen": -90.40399169921875,
+      "logps/rejected": -161.551025390625,
+      "loss": 1.3575,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": 0.019125938415527344,
+      "rewards/margins": 0.029542922973632812,
+      "rewards/rejected": -0.010416984558105469,
+      "step": 99
+    },
+    {
+      "epoch": 0.005374754776813308,
+      "grad_norm": 9.5785493850708,
+      "learning_rate": 5e-07,
+      "logits/chosen": -0.32433757185935974,
+      "logits/rejected": -0.9152166843414307,
+      "logps/chosen": -87.89921569824219,
+      "logps/rejected": -110.95304870605469,
+      "loss": 1.3612,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": 0.004671669099479914,
+      "rewards/margins": 0.02547921985387802,
+      "rewards/rejected": -0.020807553082704544,
+      "step": 100
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 1000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2209cbf6cb6a4030aa503ccb5abb4d7eb7fdc025635e1df4f63bb6e387ee6a0
+size 7096