readme: ---
  license: apache-2.0
  tags:
  - generated_from_trainer
  datasets:
  - NewSHead
  model-index:
  - name: Centrum
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # Centrum
  
  Centrum is a pretrained model for multi-document summarization, trained with centroid-based pretraining objective on the NewSHead dataset. It is initialized from [allenai/led-base-16384](https://huggingface.co/allenai/led-base-16384). The details of the approach are mentioned in the preprint [Multi-Document Summarization with Centroid-Based Pretraining](https://arxiv.org/abs/2208.01006) (Ratish Puduppully and Mark Steedman).
  It achieves the following results on the evaluation set:
  - Loss: 3.5568
  
  ## Model description
  
  The script for training and inference of Centrum is available on https://github.com/ratishsp/centrum
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 3e-05
  - train_batch_size: 1
  - eval_batch_size: 4
  - seed: 42
  - distributed_type: multi-GPU
  - num_devices: 4
  - gradient_accumulation_steps: 4
  - total_train_batch_size: 16
  - total_eval_batch_size: 16
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 10000
  - training_steps: 100000
  - mixed_precision_training: Native AMP
  - label_smoothing_factor: 0.1
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 4.1628        | 0.05  | 500    | 4.0732          |
  | 4.0278        | 0.09  | 1000   | 3.9800          |
  | 4.0008        | 0.14  | 1500   | 3.9283          |
  | 3.9564        | 0.19  | 2000   | 3.8941          |
  | 3.9193        | 0.23  | 2500   | 3.8780          |
  | 3.9185        | 0.28  | 3000   | 3.8501          |
  | 3.8881        | 0.32  | 3500   | 3.8334          |
  | 3.8869        | 0.37  | 4000   | 3.8211          |
  | 3.876         | 0.42  | 4500   | 3.8057          |
  | 3.8552        | 0.46  | 5000   | 3.7954          |
  | 3.8198        | 0.51  | 5500   | 3.7861          |
  | 3.8016        | 0.56  | 6000   | 3.7750          |
  | 3.8033        | 0.6   | 6500   | 3.7651          |
  | 3.7927        | 0.65  | 7000   | 3.7528          |
  | 3.7978        | 0.7   | 7500   | 3.7429          |
  | 3.7727        | 0.74  | 8000   | 3.7367          |
  | 3.7634        | 0.79  | 8500   | 3.7275          |
  | 3.7395        | 0.83  | 9000   | 3.7158          |
  | 3.7432        | 0.88  | 9500   | 3.7066          |
  | 3.7623        | 0.93  | 10000  | 3.7039          |
  | 3.7182        | 0.97  | 10500  | 3.6904          |
  | 3.7146        | 1.02  | 11000  | 3.6881          |
  | 3.681         | 1.07  | 11500  | 3.6797          |
  | 3.6745        | 1.11  | 12000  | 3.6750          |
  | 3.6794        | 1.16  | 12500  | 3.6748          |
  | 3.6802        | 1.21  | 13000  | 3.6696          |
  | 3.665         | 1.25  | 13500  | 3.6609          |
  | 3.6516        | 1.3   | 14000  | 3.6633          |
  | 3.6577        | 1.34  | 14500  | 3.6573          |
  | 3.6409        | 1.39  | 15000  | 3.6519          |
  | 3.6691        | 1.44  | 15500  | 3.6490          |
  | 3.6521        | 1.48  | 16000  | 3.6475          |
  | 3.6435        | 1.53  | 16500  | 3.6465          |
  | 3.6466        | 1.58  | 17000  | 3.6392          |
  | 3.644         | 1.62  | 17500  | 3.6419          |
  | 3.6347        | 1.67  | 18000  | 3.6347          |
  | 3.6205        | 1.71  | 18500  | 3.6328          |
  | 3.6451        | 1.76  | 19000  | 3.6310          |
  | 3.6327        | 1.81  | 19500  | 3.6284          |
  | 3.6166        | 1.85  | 20000  | 3.6267          |
  | 3.622         | 1.9   | 20500  | 3.6212          |
  | 3.6164        | 1.95  | 21000  | 3.6199          |
  | 3.6178        | 1.99  | 21500  | 3.6201          |
  | 3.5892        | 2.04  | 22000  | 3.6201          |
  | 3.5855        | 2.09  | 22500  | 3.6221          |
  | 3.5658        | 2.13  | 23000  | 3.6193          |
  | 3.5916        | 2.18  | 23500  | 3.6144          |
  | 3.5767        | 2.22  | 24000  | 3.6101          |
  | 3.5809        | 2.27  | 24500  | 3.6115          |
  | 3.5561        | 2.32  | 25000  | 3.6110          |
  | 3.5831        | 2.36  | 25500  | 3.6080          |
  | 3.5551        | 2.41  | 26000  | 3.6121          |
  | 3.5588        | 2.46  | 26500  | 3.6072          |
  | 3.5645        | 2.5   | 27000  | 3.6056          |
  | 3.5804        | 2.55  | 27500  | 3.6038          |
  | 3.5712        | 2.6   | 28000  | 3.6052          |
  | 3.5494        | 2.64  | 28500  | 3.6014          |
  | 3.582         | 2.69  | 29000  | 3.5995          |
  | 3.5487        | 2.73  | 29500  | 3.6051          |
  | 3.5709        | 2.78  | 30000  | 3.5954          |
  | 3.5546        | 2.83  | 30500  | 3.5941          |
  | 3.5525        | 2.87  | 31000  |