transformers tensorboard safetensors t5 text2text-generation generated_from_trainer en dataset:ccdv/arxiv-summarization base_model:google-t5/t5-small base_model:finetune:google-t5/t5-small license:apache-2.0 autotrain_compatible text-generation-inference endpoints_compatible region:us
---
base_model: t5-small
datasets:
- ccdv/arxiv-summarization
language:
- en
library_name: transformers
license: apache-2.0
metrics:
- rouge
- accuracy
tags:
- generated_from_trainer
model-index:
- name: T5_Fine_Tuned_on_Arxiv_Dataset
  results: []
---


# T5_Fine_Tuned_on_Arxiv_Dataset


## Model Description
This model is a fine-tuned version of [t5-small](https://huggingface.co/t5-small) designed for summarizing research papers from the Arxiv dataset. It utilizes an abstractive summarization approach to generate concise summaries that capture the main findings and contributions of the papers, facilitating easier understanding of complex academic content.

## Evaluation
It achieves the following results on the evaluation set:
- Loss: 2.7599
- Rouge1: 0.1635
- Rouge2: 0.0548
- Rougel: 0.1311
- Rougelsum: 0.1311
- Generated Length: 18.9852

## Model Overview
- **Model Name**: Arxiv Summarization Model
- **Model Type**: Summarization (Abstractive)
- **Version**: 1.0
- **Date**: [28-Sep-2024]
- **Authors**: Muhammad Ibtisam Afzal
- **Contact Information**: ibtisamafzal001@gmail.com

## Dataset
- **Dataset Name**: ccdv/arxiv-summarization
- **Dataset Description**: This dataset consists of articles from the Arxiv repository, paired with their respective abstracts. It is intended for training and evaluating summarization models in the academic domain.
- **Training/Validation/Test Split**: The dataset was split into training (80%), validation (10%), and test (10%) sets.
- **Data Source**: Hugging Face Datasets Hub
  
## Limitations
The model may struggle with highly technical content or specialized jargon that is not well-represented in the training dataset. Additionally, it may produce summaries that lack coherence or completeness for particularly long documents.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Generated Length |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:----------------:|
| No log        | 1.0   | 305  | 2.8130          | 0.1569 | 0.05   | 0.1256 | 0.1255    | 18.9852          |
| 3.0803        | 2.0   | 610  | 2.7704          | 0.1634 | 0.0546 | 0.1312 | 0.1311    | 18.9852          |
| 3.0803        | 3.0   | 915  | 2.7599          | 0.1635 | 0.0548 | 0.1311 | 0.1311    | 18.9852          |


### Framework versions

- Transformers 4.44.2
- Pytorch 2.4.1+cu121
- Datasets 3.0.1
- Tokenizers 0.19.1

## Acknowledgments
Thanks to Hugging Face for providing the infrastructure and datasets necessary for developing and evaluating this model.

