transformers safetensors llama text-generation summarization generated_from_trainer base_model:TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T base_model:finetune:TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T license:apache-2.0 autotrain_compatible text-generation-inference endpoints_compatible region:us
---
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
license: apache-2.0
tags:
- summarization
- generated_from_trainer
model-index:
- name: sft-sum-chosen-10lp-shuff-full-tiny
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# sft-sum-chosen-10lp-shuff-full-tiny

This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) on the martimfasantos/openai-summarize-tldr dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9409
- Nll Loss: 1.9409
- Logps/best: -72.8478
- Rewards/chosen: 2.0114
- Rewards/rejected: -0.4229
- Rewards/accuracies: 0.9998
- Rewards/margins: 2.4343
- Logps/rejected: -11.6536
- Logps/chosen: -72.8478
- Logits/rejected: -2.6479
- Logits/chosen: -2.9522

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 16
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Nll Loss | Logps/best | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:------:|:----:|:---------------:|:--------:|:----------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 2.3573        | 0.0137 | 100  | 2.3703          | 2.3703   | -88.8140   | 0.4147         | 0.0412           | 1.0                | 0.3735          | -7.0125        | -88.8140     | -2.6551         | -2.9658       |
| 2.1904        | 0.0274 | 200  | 2.1322          | 2.1322   | -79.9647   | 1.2997         | 0.0373           | 1.0                | 1.2624          | -7.0516        | -79.9647     | -2.6656         | -2.9758       |
| 1.9956        | 0.0411 | 300  | 2.0629          | 2.0629   | -77.3844   | 1.5577         | -0.1097          | 0.9995             | 1.6674          | -8.5217        | -77.3844     | -2.6813         | -2.9915       |
| 2.0379        | 0.0548 | 400  | 2.0405          | 2.0405   | -76.5483   | 1.6413         | -0.1759          | 0.9994             | 1.8173          | -9.1840        | -76.5483     | -2.6918         | -3.0033       |
| 1.9476        | 0.0685 | 500  | 2.0250          | 2.0250   | -75.9762   | 1.6985         | -0.1561          | 0.9991             | 1.8546          | -8.9858        | -75.9762     | -2.6981         | -3.0089       |
| 2.0151        | 0.0822 | 600  | 2.0134          | 2.0133   | -75.5465   | 1.7415         | -0.1979          | 0.9991             | 1.9394          | -9.4039        | -75.5465     | -2.6956         | -3.0066       |
| 1.9972        | 0.0960 | 700  | 2.0037          | 2.0037   | -75.1909   | 1.7770         | -0.2110          | 0.9997             | 1.9881          | -9.5345        | -75.1909     | -2.6886         | -2.9996       |
| 1.9851        | 0.1097 | 800  | 1.9950          | 1.9950   | -74.8615   | 1.8100         | -0.2127          | 0.9997             | 2.0226          | -9.5511        | -74.8615     | -2.6861         | -2.9971       |
| 2.0271        | 0.1234 | 900  | 1.9890          | 1.9890   | -74.6372   | 1.8324         | -0.2530          | 0.9995             | 2.0854          | -9.9543        | -74.6372     | -2.6818         | -2.9925       |
| 2.0501        | 0.1371 | 1000 | 1.9845          | 1.9845   | -74.4788   | 1.8483         | -0.3242          | 0.9997             | 2.1724          | -10.6661       | -74.4788     | -2.6491         | -2.9545       |
| 1.9699        | 0.1508 | 1100 | 1.9813          | 1.9812   | -74.3528   | 1.8609         | -0.3208          | 0.9997             | 2.1817          | -10.6327       | -74.3528     | -2.6664         | -2.9755       |
| 1.9448        | 0.1645 | 1200 | 1.9773          | 1.9772   | -74.2031   | 1.8758         | -0.2738          | 0.9997             | 2.1496          | -10.1623       | -74.2031     | -2.6739         | -2.9842       |
| 1.9606        | 0.1782 | 1300 | 1.9746          | 1.9746   | -74.0931   | 1.8868         | -0.3353          | 0.9997             | 2.2221          | -10.7775       | -74.0931     | -2.6755         | -2.9850       |
| 1.8795        | 0.1919 | 1400 | 1.9716          | 1.9715   | -73.9887   | 1.8973         | -0.3115          | 0.9997             | 2.2088          | -10.5398       | -73.9887     | -2.6658         | -2.9741       |
| 1.9585        | 0.2056 | 1500 | 1.9703          | 1.9703   | -73.9430   | 1.9018         | -0.3353          | 0.9997             | 2.2371          | -10.7774       | -73.9430     | -2.6721         | -2.9814       |
| 1.9508        | 0.2193 | 1600 | 1.9664          | 1.9664   | -73.7942   | 1.9167         | -0.4138          | 0.9998             | 2.3305          | -11.5624       | -73.7942     | -2.6751         | -2.9840       |
| 1.9041        | 0.2330 | 1700 | 1.9657          | 1.9656   | -73.7736   | 1.9188         | -0.3353          | 0.9997             | 2.2541          | -10.7776       | -73.7736     | -2.6703         | -2.9794       |
| 1.9507        | 0.2467 | 1800 | 1.9634          | 1.9634   | -73.6847   | 1.9277         | -0.3964          | 0.9998             | 2.3240          | -11.3880       | -73.6847     | -2.6728         | -2.9810       |
| 1.8942        | 0.2604 | 1900 | 1.9620          | 1.9620   | -73.6314   | 1.9330         | -0.3368          | 0.9998             | 2.2698          | -10.7926       | -73.6314     | -2.6631         | -2.9695       |
| 2.0088        | 0.2742 | 2000 | 1.9604          | 1.9603   | -73.5703   | 1.9391         | -0.3303          | 0.9997             | 2.2694          | -10.7277       | -73.5703     | -2.6651         | -2.9720       |
| 2.0277        | 0.2879 | 2100 | 1.9596          | 1.9596   | -73.5404   | 1.9421         | -0.3122          | 0.9997             | 2.2543          | -10.5463       | -73.5404     | -2.6687         | -2.9765       |
| 1.9697        | 0.3016 | 2200 | 1.9578          | 1.9578   | -73.4823   | 1.9479         | -0.3187          | 0.9998             | 2.2666          | -10.6117       | -73.4823     | -2.6615         | -2.9674       |
| 1.9756        | 0.3153 | 2300 | 1.9564          | 1.9564   | -73.4282   | 1.9533         | -0.3217          | 0.9997             | 2.2750          | -10.6410       | -73.4282     | -2.6624         | -2.9692       |
| 1.9471        | 0.3290 | 2400 | 1.9552          | 1.9551   | -73.3780   | 1.9583         | -0.3660          | 0.9997             | 2.3244          | -11.0849       | -73.3780     | -2.6636         | -2.9703       |
| 1.9646        | 0.3427 | 2500 | 1.9546          | 1.9546   | -73.3608   | 1.9601         | -0.3453          | 0.9997             | 2.3054          | -10.8779       | -73.3608     | -2.6522         | -2.9582       |
| 2.0034        | 0.3564 | 2600 | 1.9536          | 1.9536   | -73.3221   | 1.9639         | -0.4025          | 0.9998             | 2.3665          | -11.4498       | -73.3221     | -2.6635         | -2.9708       |
| 1.9853        | 0.3701 | 2700 | 1.9522          | 1.9522   | -73.2647   | 1.9697         | -0.3826          | 0.9998             | 2.3523          | -11.2507       | -73.2647     | -2.6548         | -2.9612       |
| 1.9648        | 0.3838 | 2800 | 1.9518          | 1.9518   | -73.2540   | 1.9707         | -0.4008          | 0.9998             | 2.3716          | -11.4329       | -73.2540     | -2.6557         | -2.9618       |
| 1.992         | 0.3975 | 2900 | 1.9514          | 1.9513   | -73.2347   | 1.9727         | -0.3741          | 0.9998             | 2.3468          | -11.1657       | -73.2347     | -2.6585         | -2.9649       |
| 1.9098        | 0.4112 | 3000 | 1.9501          | 1.9501   | -73.1879   | 1.9773         | -0.3653          | 0.9998             | 2.3426          | -11.0774       | -73.1879     | -2.6623         | -2.9691       |
| 2.0089        | 0.4249 | 3100 | 1.9496          | 1.9496   | -73.1694   | 1.9792         | -0.3960          | 0.9998             | 2.3752          | -11.3848       | -73.1694     | -2.6570         | -2.9627       |
| 2.0138        | 0.4386 | 3200 | 1.9487          | 1.9487   | -73.1364   | 1.9825         | -0.3799          | 0.9998             | 2.3624          | -11.2233       | -73.1364     | -2.6524         | -2.9576       |
| 1.9295        | 0.4524 | 3300 | 1.9489          | 1.9489   | -73.1488   | 1.9813         | -0.3977          | 0.9998             | 2.3790          | -11.4018       | -73.1488     | -2.6569         | -2.9628       |
| 1.9276        | 0.4661 | 3400 | 1.9479          | 1.9479   | -73.1079   | 1.9853         | -0.3945          | 0.9998             | 2.3799          | -11.3697       | -73.1079     | -2.6537         | -2.9590       |
| 1.9594        | 0.4798 | 3500 | 1.9472          | 1.9472   | -73.0821   | 1.9879         | -0.4255          | 0.9998             | 2.4135          | -11.6798       | -73.0821     | -2.6542         | -2.9600       |
| 1.9141        | 0.4935 | 3600 | 1.9471          | 1.9471   | -73.0800   | 1.9881         | -0.4024          | 0.9998             | 2.3906          | -11.4487       | -73.0800     | -2.6500         | -2.9555       |
| 1.8611        | 0.5072 | 3700 | 1.9460          | 1.9460   | -73.0338   | 1.9928         | -0.3865          | 0.9998             | 2.3793          | -11.2897       | -73.0338     | -2.6542         | -2.9599       |
| 1.8907        | 0.5209 | 3800 | 1.9460          | 1.9460   | -73.0372   | 1.9924         | -0.3918          | 0.9998             | 2.3843          | -11.3429       | -73.0372     | -2.6504         | -2.9556       |
| 1.9147        | 0.5346 | 3900 | 1.9456          | 1.9456   | -73.0218   | 1.9940         | -0.3939          | 0.9998             | 2.3879          | -11.3637       | -73.0218     | -2.6498         | -2.9550       |
| 1.9485        | 0.5483 | 4000 | 1.9454          | 1.9454   | -73.0146   | 1.9947         | -0.4036          | 0.9998             | 2.3983          | -11.4605       | -73.0146     | -2.6513         | -2.9565       |
| 1.9379        | 0.5620 | 4100 | 1.9448          | 1.9448   | -72.9908   | 1.9971         | -0.3932          | 0.9998             | 2.3902          | -11.3561       | -72.9908     | -2.6501         | -2.9550       |
| 1.8956        | 0.5757 | 4200 | 1.9444          | 1.9443   | -72.9738   | 1.9988         | -0.4097          | 0.9998             | 2.4084          | -11.5214       | -72.9738     | -2.6477         | -2.9518       |
| 1.9916        | 0.5894 | 4300 | 1.9440          | 1.9440   | -72.9580   | 2.0003         | -0.4049          | 0.9998             | 2.4053          | -11.4737       | -72.9580     | -2.6473         | -2.9514       |
| 1.8885        | 0.6031 | 4400 | 1.9441          | 1.9441   | -72.9673   | 1.9994         | -0.3808          | 0.9998             | 2.3802          | -11.2320       | -72.9673     | -2.6464         | -2.9503       |
| 1.9078        | 0.6169 | 4500 | 1.9437          | 1.9436   | -72.9481   | 2.0013         | -0.4206          | 0.9998             | 2.4220          | -11.6308       | -72.9481     | -2.6465         | -2.9503       |
| 1.9037        | 0.6306 | 4600 | 1.9435          | 1.9434   | -72.9426   | 2.0019         | -0.3718          | 0.9998             | 2.3737          | -11.1427       | -72.9426     | -2.6441         | -2.9481       |
| 1.9558        | 0.6443 | 4700 | 1.9427          | 1.9427   | -72.9121   | 2.0049         | -0.3758          | 0.9998             | 2.3807          | -11.1827       | -72.9121     | -2.6445         | -2.9484       |
| 1.9416        | 0.6580 | 4800 | 1.9429          | 1.9428   | -72.9187   | 2.0043         | -0.3698          | 0.9998             | 2.3741          | -11.1227       | -72.9187     | -2.6447         | -2.9486       |
| 1.9471        | 0.6717 | 4900 | 1.9427          | 1.9427   | -72.9167   | 2.0045         | -0.4041          | 0.9998             | 2.4085          | -11.4650       | -72.9167     | -2.6447         | -2.9486       |
| 1.9237        | 0.6854 | 5000 | 1.9425          | 1.9425   | -72.9062   | 2.0055         | -0.4023          | 0.9998             | 2.4079          | -11.4479       | -72.9062     | -2.6451         | -2.9490       |
| 1.9687        | 0.6991 | 5100 | 1.9422          | 1.9421   | -72.8930   | 2.0068         | -0.4106          | 0.9998             | 2.4174          | -11.5306       | -72.8930     | -2.6475         | -2.9516       |
| 1.9274        | 0.7128 | 5200 | 1.9420          | 1.9420   | -72.8846   | 2.0077         | -0.3934          | 0.9998             | 2.4011          | -11.3589       | -72.8846     | -2.6454         | -2.9492       |
| 1.8258        | 0.7265 | 5300 | 1.9418          | 1.9418   | -72.8788   | 2.0083         | -0.3905          | 0.9998             | 2.3987          | -11.3293       | -72.8788     | -2.6458         | -2.9498       |
| 1.8978        | 0.7402 | 5400 | 1.9416          | 1.9416   | -72.8710   | 2.0090         | -0.4199          | 0.9998             | 2.4289          | -11.6232       | -72.8710     | -2.6475         | -2.9515       |
| 1.9706        | 0.7539 | 5500 | 1.9416          | 1.9416   | -72.8733   | 2.0088         | -0.4296          | 0.9998             | 2.4384          | -11.7202       | -72.8733     | -2.6467         | -2.9506       |
| 1.8711        | 0.7676 | 5600 | 1.9416          | 1.9415   | -72.8708   | 2.0091         | -0.4093          | 0.9998             | 2.4183          | -11.5174       | -72.8708     | -2.6454         | -2.9492       |
| 1.925         | 0.7813 | 5700 | 1.9412          | 1.9411   | -72.8550   | 2.0106         | -0.4237          | 0.9998             | 2.4344          | -11.6619       | -72.8550     | -2.6463         | -2.9502       |
| 1.952         | 0.7951 | 5800 | 1.9412          | 1.9411   | -72.8554   | 2.0106         | -0.4179          | 0.9998             | 2.4285          | -11.6032       | -72.8554     | -2.6463         | -2.9503       |
| 1.9295        | 0.8088 | 5900 | 1.9413          | 1.9413   | -72.8621   | 2.0099         | -0.4133          | 0.9998             | 2.4233          | -11.5578       | -72.8621     | -2.6463         | -2.9503       |
| 1.9457        | 0.8225 | 6000 | 1.9413          | 1.9413   | -72.8636   | 2.0098         | -0.4083          | 0.9998             | 2.4180          | -11.5072       | -72.8636     | -2.6459         | -2.9499       |
| 1.9016        | 0.8362 | 6100 | 1.9412          | 1.9412   | -72.8592   | 2.0102         | -0.4150          | 0.9998             | 2.4252          | -11.5748       | -72.8592     | -2.6471         | -2.9513       |
| 1.9789        | 0.8499 | 6200 | 1.9413          | 1.9413   | -72.8632   | 2.0098         | -0.4221          | 0.9998             | 2.4319          | -11.6458       | -72.8632     | -2.6477         | -2.9520       |
| 1.944         | 0.8636 | 6300 | 1.9411          | 1.9411   | -72.8542   | 2.0107         | -0.4232          | 0.9998             | 2.4339          | -11.6568       | -72.8542     | -2.6475         | -2.9518       |
| 1.9435        | 0.8773 | 6400 | 1.9410          | 1.9409   | -72.8496   | 2.0112         | -0.4278          | 0.9998             | 2.4390          | -11.7027       | -72.8496     | -2.6479         | -2.9523       |
| 1.917         | 0.8910 | 6500 | 1.9410          | 1.9410   | -72.8519   | 2.0109         | -0.4237          | 0.9998             | 2.4346          | -11.6610       | -72.8519     | -2.6482         | -2.9525       |
| 1.9243        | 0.9047 | 6600 | 1.9410          | 1.9410   | -72.8520   | 2.0109         | -0.4202          | 0.9998             | 2.4311          | -11.6265       | -72.8520     | -2.6480         | -2.9523       |
| 1.8624        | 0.9184 | 6700 | 1.9409          | 1.9409   | -72.8485   | 2.0113         | -0.4202          | 0.9998             | 2.4314          | -11.6260       | -72.8485     | -2.6477         | -2.9520       |
| 1.8998        | 0.9321 | 6800 | 1.9410          | 1.9409   | -72.8489   | 2.0112         | -0.4227          | 0.9998             | 2.4340          | -11.6518       | -72.8489     | -2.6478         | -2.9521       |
| 1.9654        | 0.9458 | 6900 | 1.9410          | 1.9409   | -72.8490   | 2.0112         | -0.4228          | 0.9998             | 2.4341          | -11.6529       | -72.8490     | -2.6478         | -2.9521       |
| 1.9113        | 0.9595 | 7000 | 1.9409          | 1.9409   | -72.8471   | 2.0114         | -0.4228          | 0.9998             | 2.4342          | -11.6520       | -72.8471     | -2.6477         | -2.9520       |
| 1.951         | 0.9733 | 7100 | 1.9410          | 1.9410   | -72.8501   | 2.0111         | -0.4228          | 0.9998             | 2.4339          | -11.6524       | -72.8501     | -2.6478         | -2.9521       |
| 1.9863        | 0.9870 | 7200 | 1.9409          | 1.9409   | -72.8478   | 2.0114         | -0.4229          | 0.9998             | 2.4343          | -11.6536       | -72.8478     | -2.6479         | -2.9522       |


### Framework versions

- Transformers 4.41.2
- Pytorch 2.1.2
- Datasets 2.20.0
- Tokenizers 0.19.1
