transformers safetensors t5 text2text-generation generated_from_trainer dataset:code_search_net model-index autotrain_compatible text-generation-inference endpoints_compatible region:us
---
datasets:
- code_search_net
metrics:
- bleu
tags:
- generated_from_trainer
model-index:
- name: pretrain_base_tokenizer
  results:
  - task:
      type: text2text-generation
      name: Sequence-to-sequence Language Modeling
    dataset:
      name: code_search_net
      type: code_search_net
      config: python
      split: test
      args: python
    metrics:
    - type: bleu
      value: 0.07449739698581556
      name: Bleu
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# pretrain_base_tokenizer

This model was trained from scratch on the code_search_net dataset.
It achieves the following results on the evaluation set:
- Loss: 2.1008
- Bleu: 0.0745
- Precisions: [0.370227852188713, 0.13803247473556413, 0.07398987834019316, 0.04421999242711094]
- Brevity Penalty: 0.6551
- Length Ratio: 0.7028
- Translation Length: 594596
- Reference Length: 846059

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20

### Training results

| Training Loss | Epoch | Step   | Bleu   | Brevity Penalty | Length Ratio | Validation Loss | Precisions                                                                            | Reference Length | Translation Length |
|:-------------:|:-----:|:------:|:------:|:---------------:|:------------:|:---------------:|:-------------------------------------------------------------------------------------:|:----------------:|:------------------:|
| 2.4112        | 1.0   | 25762  | 0.0658 | 0.6617          | 0.7078       | 2.3310          | [0.35249982048117723, 0.12327087650542343, 0.06319307060145318, 0.03566263068721848]  | 846059           | 598823             |
| 2.3334        | 2.0   | 51524  | 0.0681 | 0.6617          | 0.7078       | 2.2582          | [0.35832782242172834, 0.127192419612726, 0.06572103592555187, 0.03742664572798245]    | 846059           | 598812             |
| 2.2441        | 3.0   | 77286  | 0.0696 | 0.6529          | 0.7011       | 2.2180          | [0.36256557741844125, 0.13175932444407085, 0.06865854925017445, 0.039288874037902925] | 846059           | 593192             |
| 2.1798        | 4.0   | 103048 | 0.0721 | 0.6729          | 0.7162       | 2.1907          | [0.36450348611741407, 0.13225409723744794, 0.06903673685953533, 0.0396539707725045]   | 846059           | 605975             |
| 2.1424        | 5.0   | 128810 | 0.0715 | 0.6561          | 0.7035       | 2.1736          | [0.3668020289217111, 0.13425532471838544, 0.07022789074894986, 0.04068744822577534]   | 846059           | 595193             |
| 2.1132        | 6.0   | 154572 | 0.0739 | 0.6875          | 0.7275       | 2.1539          | [0.36025300866163096, 0.13232255476642318, 0.06955911290379053, 0.040195441044440436] | 846059           | 615473             |
| 2.0984        | 7.0   | 180334 | 0.0721 | 0.6587          | 0.7055       | 2.1471          | [0.36612131721578584, 0.13431329561035363, 0.0708157263719857, 0.041272288902252124]  | 846059           | 596865             |
| 2.0785        | 8.0   | 206096 | 0.0724 | 0.6756          | 0.7183       | 2.1353          | [0.36380808213768595, 0.13209779987841613, 0.06888583628832168, 0.039797612956022015] | 846059           | 607760             |
| 2.044         | 9.0   | 231858 | 0.0651 | 0.5890          | 0.6539       | 2.1307          | [0.3747329223983659, 0.13597984423722942, 0.07083334152049311, 0.041232475735633954]  | 846059           | 553210             |
| 2.0022        | 10.0  | 257620 | 0.0678 | 0.6182          | 0.6752       | 2.1244          | [0.37057115300122706, 0.13501863826827087, 0.07054691458053057, 0.04094138244503552]  | 846059           | 571283             |
| 2.0115        | 11.0  | 283382 | 0.0714 | 0.6437          | 0.6942       | 2.1181          | [0.3696569336851962, 0.1359002395637604, 0.07172057187893609, 0.04198004369041997]    | 846059           | 587350             |
| 1.9957        | 12.0  | 309144 | 0.0780 | 0.7340          | 0.7638       | 2.1182          | [0.3562361599633563, 0.13051385463885645, 0.06873666863799165, 0.039808098889084334]  | 846059           | 646223             |
| 1.9816        | 13.0  | 334906 | 0.0748 | 0.6775          | 0.7198       | 2.1112          | [0.3644272643077186, 0.1348813193666958, 0.07171355661769002, 0.04221956829440906]    | 846059           | 608972             |
| 1.9799        | 14.0  | 360668 | 0.0729 | 0.6567          | 0.7039       | 2.1094          | [0.3683080239907046, 0.1360146909050323, 0.07189528256366162, 0.04211029597965069]    | 846059           | 595564             |
| 1.9721        | 15.0  | 386430 | 0.0724 | 0.6428          | 0.6935       | 2.1035          | [0.37174066063670774, 0.13775257176864, 0.07323700636731323, 0.042981616643797974]    | 846059           | 586737             |
| 1.9415        | 16.0  | 412192 | 0.0707 | 0.6275          | 0.6822       | 2.1052          | [0.37395952455210174, 0.1379846553581918, 0.07303615398474567, 0.04286080713028393]   | 846059           | 577140             |
| 1.921         | 17.0  | 437954 | 0.0755 | 0.6693          | 0.7135       | 2.1031          | [0.368775375991227, 0.137205943045811, 0.07336018463397673, 0.04358297628398173]      | 846059           | 603671             |
| 1.9281        | 18.0  | 463716 | 0.0730 | 0.6426          | 0.6934       | 2.1008          | [0.3719142378879256, 0.13830769094500395, 0.0738741230374939, 0.043850156367431746]   | 846059           | 586646             |
| 1.9619        | 19.0  | 489478 | 0.0741 | 0.6539          | 0.7019       | 2.1011          | [0.3690046967173499, 0.1375453499071136, 0.07371470581812624, 0.044035281313872486]   | 846059           | 593819             |
| 1.9177        | 20.0  | 515240 | 2.1008 | 0.0745          | [0.370227852188713, 0.13803247473556413, 0.07398987834019316, 0.04421999242711094]| 0.6551          | 0.7028                                                                                | 594596           | 846059             |


### Framework versions

- Transformers 4.37.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2
