transformers pytorch bart text2text-generation generated_from_trainer dataset:clupubhealth base_model:facebook/bart-large base_model:finetune:facebook/bart-large license:apache-2.0 model-index autotrain_compatible endpoints_compatible region:us
---
base_model: facebook/bart-large
datasets:
- clupubhealth
license: apache-2.0
metrics:
- rouge
tags:
- generated_from_trainer
model-index:
- name: bart-pubhealth-expanded
  results:
  - task:
      type: text2text-generation
      name: Sequence-to-sequence Language Modeling
    dataset:
      name: clupubhealth
      type: clupubhealth
      config: expanded
      split: test
      args: expanded
    metrics:
    - type: rouge
      value: 29.8528
      name: Rouge1
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bart-pubhealth-expanded

This model is a fine-tuned version of [facebook/bart-large](https://huggingface.co/facebook/bart-large) on the clupubhealth dataset.
It achieves the following results on the evaluation set:
- Loss: 2.3926
- Rouge1: 29.8528
- Rouge2: 10.8495
- Rougel: 23.3682
- Rougelsum: 23.7565
- Gen Len: 19.85

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 2.7469        | 0.26  | 500   | 2.0845          | 30.9611 | 10.7145 | 23.9719 | 24.1042   | 19.905  |
| 2.5524        | 0.51  | 1000  | 2.0628          | 32.0352 | 11.8898 | 24.9032 | 25.1368   | 19.895  |
| 2.429         | 0.77  | 1500  | 2.0787          | 32.2632 | 12.0353 | 25.1245 | 25.3728   | 19.895  |
| 2.2234        | 1.03  | 2000  | 2.1178          | 30.6437 | 11.5713 | 24.9071 | 25.1126   | 19.955  |
| 2.1249        | 1.29  | 2500  | 2.1183          | 31.6095 | 11.6573 | 25.0593 | 25.2063   | 19.87   |
| 2.0302        | 1.54  | 3000  | 2.1319          | 30.7417 | 11.4924 | 24.6388 | 24.8722   | 19.895  |
| 1.9761        | 1.8   | 3500  | 2.1850          | 31.6709 | 11.3036 | 24.4853 | 24.7571   | 19.87   |
| 1.8279        | 2.06  | 4000  | 2.2092          | 31.5778 | 11.59   | 24.7599 | 24.9956   | 19.825  |
| 1.8083        | 2.32  | 4500  | 2.1781          | 31.0441 | 10.7513 | 24.0656 | 24.3112   | 19.89   |
| 1.7527        | 2.57  | 5000  | 2.2155          | 31.1191 | 11.4519 | 24.4673 | 24.7157   | 19.81   |
| 1.723         | 2.83  | 5500  | 2.2024          | 31.9787 | 12.3158 | 24.9863 | 25.2597   | 19.94   |
| 1.5975        | 3.09  | 6000  | 2.2567          | 31.236  | 10.9733 | 24.1302 | 24.3433   | 19.9    |
| 1.5933        | 3.35  | 6500  | 2.2425          | 31.022  | 11.0249 | 24.1257 | 24.3555   | 19.92   |
| 1.5792        | 3.6   | 7000  | 2.2428          | 29.8844 | 10.3622 | 23.0802 | 23.4003   | 19.96   |
| 1.5718        | 3.86  | 7500  | 2.2367          | 31.2369 | 11.3854 | 24.8528 | 25.1287   | 19.815  |
| 1.4467        | 4.12  | 8000  | 2.2988          | 30.4903 | 10.4057 | 23.9914 | 24.239    | 19.715  |
| 1.4458        | 4.37  | 8500  | 2.2738          | 31.4345 | 11.2989 | 24.4239 | 24.6047   | 19.75   |
| 1.4342        | 4.63  | 9000  | 2.3092          | 28.8421 | 10.5744 | 23.0084 | 23.1741   | 19.855  |
| 1.4416        | 4.89  | 9500  | 2.2747          | 31.7111 | 11.5903 | 24.3422 | 24.6867   | 19.945  |
| 1.3437        | 5.15  | 10000 | 2.3203          | 31.11   | 11.0    | 24.6098 | 24.7362   | 19.81   |
| 1.3525        | 5.4   | 10500 | 2.3085          | 29.414  | 10.3412 | 23.3134 | 23.6552   | 19.935  |
| 1.3533        | 5.66  | 11000 | 2.3123          | 31.321  | 11.2686 | 23.9922 | 24.336    | 19.77   |
| 1.3248        | 5.92  | 11500 | 2.2916          | 30.8841 | 10.779  | 23.9407 | 24.0865   | 19.845  |
| 1.2617        | 6.18  | 12000 | 2.3530          | 29.7167 | 10.3162 | 23.4805 | 23.724    | 19.93   |
| 1.2846        | 6.43  | 12500 | 2.3712          | 28.3334 | 9.8425  | 22.1151 | 22.2951   | 19.92   |
| 1.2472        | 6.69  | 13000 | 2.3378          | 29.563  | 10.0033 | 23.1863 | 23.5065   | 19.865  |
| 1.2934        | 6.95  | 13500 | 2.3262          | 29.137  | 10.1232 | 22.9234 | 23.3799   | 19.855  |
| 1.2136        | 7.21  | 14000 | 2.3640          | 29.753  | 10.4865 | 23.4892 | 23.8778   | 19.885  |
| 1.2096        | 7.46  | 14500 | 2.3654          | 29.512  | 10.3891 | 23.0427 | 23.3684   | 19.88   |
| 1.211         | 7.72  | 15000 | 2.3491          | 30.9014 | 10.9117 | 24.127  | 24.3518   | 19.785  |
| 1.1954        | 7.98  | 15500 | 2.3626          | 29.0622 | 10.5405 | 22.7407 | 22.9454   | 19.84   |
| 1.1756        | 8.23  | 16000 | 2.3759          | 29.5277 | 10.2961 | 22.7888 | 23.1239   | 19.88   |
| 1.1516        | 8.49  | 16500 | 2.3772          | 29.3161 | 10.1894 | 23.0404 | 23.486    | 19.885  |
| 1.1604        | 8.75  | 17000 | 2.3710          | 29.6161 | 10.3543 | 22.8748 | 23.1849   | 19.905  |
| 1.1639        | 9.01  | 17500 | 2.3889          | 30.2817 | 10.8654 | 23.6438 | 23.8639   | 19.895  |
| 1.12          | 9.26  | 18000 | 2.3968          | 28.8747 | 9.8686  | 22.2775 | 22.6541   | 19.895  |
| 1.1136        | 9.52  | 18500 | 2.3950          | 30.1197 | 10.8992 | 23.2575 | 23.5732   | 19.86   |
| 1.1437        | 9.78  | 19000 | 2.3926          | 29.8528 | 10.8495 | 23.3682 | 23.7565   | 19.85   |


### Framework versions

- Transformers 4.31.0
- Pytorch 2.0.1+cu117
- Datasets 2.7.1
- Tokenizers 0.13.2
