transformers pytorch pegasus_x text2text-generation generated_from_trainer en dataset:arxiv-summarization arxiv:1804.05685 base_model:google/pegasus-x-base base_model:finetune:google/pegasus-x-base license:mit model-index autotrain_compatible endpoints_compatible region:us
---
base_model: google/pegasus-x-base
datasets:
- arxiv-summarization
language:
- en
license: mit
metrics:
- rouge
tags:
- generated_from_trainer
widget:
- text: '

    [Abstract] The dominant sequence transduction models are based on complex recurrent
    or convolutional neural networks in an encoder-decoder configuration. The best
    performing models also connect the encoder and decoder through an attention mechanism.
    We propose a new simple network architecture, the Transformer, based solely on
    attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments
    on two machine translation tasks show these models to be superior in quality while
    being more parallelizable and requiring significantly less time to train. Our
    model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving
    over the existing best results, including ensembles by over 2 BLEU. On the WMT
    2014 English-to-French translation task, our model establishes a new single-model
    state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs,
    a small fraction of the training costs of the best models from the literature.
    We show that the Transformer generalizes well to other tasks by applying it successfully
    to English constituency parsing both with large and limited training data. [Introduction]
    Recurrent neural networks, long short-term memory [13] and gated recurrent [7]
    neural networks in particular, have been firmly established as state of the art
    approaches in sequence modeling and transduction problems such as language modeling
    and machine translation [35, 2, 5]. Numerous efforts have since continued to push
    the boundaries of recurrent language models and encoder-decoder architectures
    [38, 24, 15]. Recurrent models typically factor computation along the symbol positions
    of the input and output sequences. Aligning the positions to steps in computation
    time, they generate a sequence of hidden states ht, as a function of the previous
    hidden state ht−1 and the input for position t. This inherently sequential nature
    precludes parallelization within training examples, which becomes critical at
    longer sequence lengths, as memory constraints limit batching across examples.
    Recent work has achieved significant improvements in computational efficiency
    through factorization tricks [21] and conditional computation [32], while also
    improving model performance in case of the latter. The fundamental constraint
    of sequential computation, however, remains. Attention mechanisms have become
    an integral part of compelling sequence modeling and transduction models in various
    tasks, allowing modeling of dependencies without regard to their distance in the
    input or output sequences [2, 19]. In all but a few cases [27], however, such
    attention mechanisms are used in conjunction with a recurrent network. In this
    work we propose the Transformer, a model architecture eschewing recurrence and
    instead relying entirely on an attention mechanism to draw global dependencies
    between input and output. The Transformer allows for significantly more parallelization
    and can reach a new state of the art in translation quality after being trained
    for as little as twelve hours on eight P100 GPUs. '
  example_title: Attention Is All You Need
- text: '[Abstract] In this work, we explore prompt tuning, a simple yet effective
    mechanism for learning soft prompts to condition frozen language models to perform
    specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft
    prompts are learned through backpropagation and can be tuned to incorporate signal
    from any number of labeled examples. Our end-to-end learned approach outperforms
    GPT-3''s few-shot learning by a large margin. More remarkably, through ablations
    on model size using T5, we show that prompt tuning becomes more competitive with
    scale: as models exceed billions of parameters, our method closes the gap and
    matches the strong performance of model tuning (where all model weights are tuned).
    This finding is especially relevant in that large models are costly to share and
    serve, and the ability to reuse one frozen model for multiple downstream tasks
    can ease this burden. Our method can be seen as a simplification of the recently
    proposed prefix tuning of Li and Liang (2021), and we provide a comparison to
    this and other similar approaches. Finally, we show that conditioning a frozen
    model with soft prompts confers benefits in robustness to domain transfer, as
    compared to full model tuning. [Introduction] With the wide success of pre-trained
    large language models, a range of techniques has arisen to adapt these general-purpose
    models to downstream tasks. ELMo (Peters et al., 2018) proposed freezing the pre-trained
    model and learning a task-specific weighting of its per-layer representations.
    However, since GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), the
    dominant adaptation technique has been model tuning (or fine-tuning), where all
    model parameters are tuned during adaptation, as proposed by Howard and Ruder
    (2018).More recently, Brown et al. (2020) showed that prompt design (or priming)
    is surprisingly effective at modulating a frozen GPT-3 model’s behavior through
    text prompts. Prompts are typically composed of a task description and/or several
    canonical examples. This return to freezing pre-trained models is appealing, especially
    as model size continues to increase. Rather than requiring a separate copy of
    the model for each downstream task, a single generalist model can simultaneously
    serve many different tasks. Unfortunately, prompt-based adaptation has several
    key drawbacks. Task description is error-prone and requires human involvement,
    and the effectiveness of a prompt is limited by how much conditioning text can
    fit into the model’s input. As a result, downstream task quality still lags far
    behind that of tuned models. For instance, GPT-3 175B fewshot performance on SuperGLUE
    is 17.5 points below fine-tuned T5-XXL (Raffel et al., 2020) (71.8 vs. 89.3) despite
    using 16 times more parameters. Several efforts to automate prompt design have
    been recently proposed. Shin et al. (2020) propose a search algorithm over the
    discrete space of words, guided by the downstream application training data. While
    this technique outperforms manual prompt design, there is still a gap relative
    to model tuning. Li and Liang (2021) propose prefix tuning and show strong results
    on generative tasks. This method freezes the model parameters and backpropagates
    the error during tuning to prefix activations prepended to each layer in the encoder
    stack, including the input layer. Hambardzumyan et al. (2021) simplify this recipe
    by restricting the trainable parameters to the input and output subnetworks of
    a masked language model, and show reasonable results on classifications tasks.
    In this paper, we propose prompt tuning as a further simplification for adapting
    language models. We freeze the entire pre-trained model and only allow an additional
    k tunable tokens per downstream task to be prepended to the input text. This soft
    prompt is trained end-to-end and can condense the signal from a full labeled dataset,
    allowing our method to outperform few-shot prompts and close the quality gap with
    model tuning (Figure 1). At the same time, since a single pre-trained model is
    recycled for all downstream tasks, we retain the efficient serving benefits of
    frozen models (Figure 2). While we developed our method concurrently with Li and
    Liang (2021) and Hambardzumyan et al. (2021), we are the first to show that prompt
    tuning alone (with no intermediate-layer prefixes or task-specific output layers)
    is sufficient to be competitive with model tuning. Through detailed experiments
    in sections 2–3, we demonstrate that language model capacity is a key ingredient
    for these approaches to succeed. As Figure 1 shows, prompt tuning becomes more
    competitive with scale. We compare with similar approaches in Section 4. Explicitly
    separating task-specific parameters from the generalist parameters needed for
    general language-understanding has a range of additional benefits. We show in
    Section 5 that by capturing the task definition in the prompt while keeping the
    generalist parameters fixed, we are able to achieve better resilience to domain
    shifts. In Section 6, we show that prompt ensembling, learning multiple prompts
    for the same task, can boost quality and is more efficient than classic model
    ensembling. Finally, in Section 7, we investigate the interpretability of our
    learned soft prompts. In sum, our key contributions are: 1. Proposing prompt tuning
    and showing its competitiveness with model tuning in the regime of large language
    models. 2. Ablating many design choices, and showing quality and robustness improve
    with scale. 3. Showing prompt tuning outperforms model tuning on domain shift
    problems. 4. Proposing prompt ensembling and showing its effectiveness. '
  example_title: PEFT (2104.08691)
- text: '[Abstract] For the first time in the world, we succeeded in synthesizing
    the room-temperature superconductor (Tc≥400 K, 127∘C) working at ambient pressure
    with a modified lead-apatite (LK-99) structure. The superconductivity of LK-99
    is proved with the Critical temperature (Tc), Zero-resistivity, Critical current
    (Ic), Critical magnetic field (Hc), and the Meissner effect. The superconductivity
    of LK-99 originates from minute structural distortion by a slight volume shrinkage
    (0.48 %), not by external factors such as temperature and pressure. The shrinkage
    is caused by Cu2+ substitution of Pb2+(2) ions in the insulating network of Pb(2)-phosphate
    and it generates the stress. It concurrently transfers to Pb(1) of the cylindrical
    column resulting in distortion of the cylindrical column interface, which creates
    superconducting quantum wells (SQWs) in the interface. The heat capacity results
    indicated that the new model is suitable for explaining the superconductivity
    of LK-99. The unique structure of LK-99 that allows the minute distorted structure
    to be maintained in the interfaces is the most important factor that LK-99 maintains
    and exhibits superconductivity at room temperatures and ambient pressure. [Introduction]  Since
    the discovery of the first superconductor(1), many efforts to search for new roomtemperature
    superconductors have been carried out worldwide(2, 3) through their experimental
    clarity or/and theoretical perspectives(4-8). The recent success of developing
    room-temperature superconductors with hydrogen sulfide(9) and yttrium super-hydride(10)
    has great attention worldwide, which is expected by strong electron-phonon coupling
    theory with high-frequency hydrogen phonon modes(11, 12). However, it is difficult
    to apply them to actual application devices in daily life because of the tremendously
    high pressure, and more efforts are being made to overcome the high-pressure problem(13).
    For the first time in the world, we report the success in synthesizing a room-temperature
    and ambient-pressure superconductor with a chemical approach to solve the temperature
    and pressure problem. We named the first room temperature and ambient pressure
    superconductor LK-99. The superconductivity of LK-99 proved with the Critical
    temperature (Tc), Zero-resistivity, Critical current (Ic), Critical magnetic field
    (Hc), and Meissner effect(14, 15). Several data were collected and analyzed in
    detail to figure out the puzzle of superconductivity of LK-99: X-ray diffraction
    (XRD), X-ray photoelectron spectroscopy (XPS), Electron Paramagnetic Resonance
    Spectroscopy (EPR), Heat Capacity, and Superconducting quantum interference device
    (SQUID) data. Henceforth in this paper, we will report and discuss our new findings
    including superconducting quantum wells associated with the superconductivity
    of LK-99.'
  example_title: LK-99 (Not NLP)
- text: '[Abstract] Abstract Evaluation practices in natural language generation (NLG)
    have many known flaws, but improved evaluation approaches are rarely widely adopted.
    This issue has become more urgent, since neural NLG models have improved to the
    point where they can often no longer be distinguished based on the surfacelevel
    features that older metrics rely on. This paper surveys the issues with human
    and automatic model evaluations and with commonly used datasets in NLG that have
    been pointed out over the past 20 years. We summarize, categorize, and discuss
    how researchers have been addressing these issues and what their findings mean
    for the current state of model evaluations. Building on those insights, we lay
    out a long-term vision for NLG evaluation and propose concrete steps for researchers
    to improve their evaluation processes. Finally, we analyze 66 NLG papers from
    recent NLP conferences in how well they already follow these suggestions and identify
    which areas require more drastic changes to the status quo. [Introduction] There
    are many issues with the evaluation of models that generate natural language.
    For example, datasets are often constructed in a way that prevents measuring tail
    effects of robustness, and they almost exclusively cover English. Most automated
    metrics measure only similarity between model output and references instead of
    fine-grained quality aspects (and even that poorly). Human evaluations have a
    high variance and, due to insufficient documentation, rarely produce replicable
    results. These issues have become more urgent as the nature of models that generate
    language has changed without significant changes to how they are being evaluated.
    While evaluation methods can capture surface-level improvements in text generated
    by state-of-the-art models (such as increased fluency) to some extent, they are
    ill-suited to detect issues with the content of model outputs, for example if
    they are not attributable to input information. These ineffective evaluations
    lead to overestimates of model capabilities. Deeper analyses uncover that popular
    models fail even at simple tasks by taking shortcuts, overfitting, hallucinating,
    and not being in accordance with their communicative goals. Identifying these
    shortcomings, many recent papers critique evaluation techniques or propose new
    ones. But almost none of the suggestions are followed or new techniques used.
    There is an incentive mismatch between conducting high-quality evaluations and
    publishing new models or modeling techniques. While general-purpose evaluation
    techniques could lower the barrier of entry for incorporating evaluation advances
    into model development, their development requires resources that are hard to
    come by, including model outputs on validation and test sets or large quantities
    of human assessments of such outputs. Moreover, some issues, like the refinement
    of datasets, require iterative processes where many researchers collaborate. All
    this leads to a circular dependency where evaluations of generation models can
    be improved only if generation models use better evaluations. We find that there
    is a systemic difference between selecting the best model and characterizing how
    good this model really is. Current evaluation techniques focus on the first, while
    the second is required to detect crucial issues. More emphasis needs to be put
    on measuring and reporting model limitations, rather than focusing on producing
    the highest performance numbers. To that end, this paper surveys analyses and
    critiques of evaluation approaches (sections 3 and 4) and of commonly used NLG
    datasets (section 5). Drawing on their insights, we describe how researchers developing
    modeling techniques can help to improve and subsequently benefit from better evaluations
    with methods available today (section 6). Expanding on existing work on model
    documentation and formal evaluation processes (Mitchell et al., 2019; Ribeiro
    et al., 2020), we propose releasing evaluation reports which focus on demonstrating
    NLG model shortcomings using evaluation suites. These reports should apply a complementary
    set of automatic metrics, include rigorous human evaluations, and be accompanied
    by data releases that allow for re-analysis with improved metrics. In an analysis
    of 66 recent EMNLP, INLG, and ACL papers along 29 dimensions related to our suggestions
    (section 7), we find that the first steps toward an improved evaluation are already
    frequently taken at an average rate of 27%. The analysis uncovers the dimensions
    that require more drastic changes in the NLG community. For example, 84% of papers
    already report results on multiple datasets and more than 28% point out issues
    in them, but we found only a single paper that contributed to the dataset documentation,
    leaving future researchers to re-identify those issues. We further highlight typical
    unsupported claims and a need for more consistent data release practices. Following
    the suggestions and results, we discuss how incorporating the suggestions can
    improve evaluation research, how the suggestions differ from similar ones made
    for NLU, and how better metrics can benefit model development itself (section
    8). '
  example_title: NLG-Eval (2202.06935)
model-index:
- name: Long-paper-summarization-pegasus-x-b
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: ccdv/arxiv-summarization
      type: ccdv/arxiv-summarization
      config: section
      split: test
      args: section
    metrics:
    - type: rouge
      value: 35.6639
      name: ROUGE-1
    - type: rouge
      value: 9.81362
      name: ROUGE-2
    - type: rouge
      value: 19.9013
      name: ROUGE-L
    - type: rouge
      value: 28.1444
      name: ROUGE-LSum
---




<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Long-paper-summarization-pegasus-x-b

This model is a fine-tuned version of [google/pegasus-x-base](https://huggingface.co/google/pegasus-x-base) on the arxiv-summarization dataset.
It achieves the following results on the evaluation set:
- Loss: 2.7262


## Model Description / Training and evaluation data


**Base Model**: [Pegasus-x-base (State-of-the-art for Long Context Summarization)](https://huggingface.co/google/pegasus-x-base)

**Finetuning Dataset**: 
- We used **train[25000:100000] of ArXiv Dataset (Cohan et al., 2018, NAACL-HLT 2018)** [[PDF]](https://arxiv.org/abs/1804.05685)
- (Full length is 200,000+, We will upload full trained Model soon)

**GPU**: (RTX A6000) x 1

**Train time**: About 24 hours for 3 epochs

**Test time**: About 8 hours for test dataset.


## Intended uses & limitations

- **Research Paper Summarization**



### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 64
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 390
- **num_epochs: 3 (takes about 24 hours)**

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 3.401         | 0.33  | 390  | 2.3985          |
| 2.5444        | 0.67  | 780  | 2.2461          |
| 2.4849        | 1.0   | 1170 | 2.2690          |
| 2.5735        | 1.33  | 1560 | 2.3334          |
| 2.7045        | 1.66  | 1950 | 2.4330          |
| 2.8939        | 2.0   | 2340 | 2.5461          |
| 3.0773        | 2.33  | 2730 | 2.6502          |
| 3.2149        | 2.66  | 3120 | 2.7039          |
| 3.2844        | 3.0   | 3510 | 2.7262          |



### Framework versions

- Transformers 4.32.1
- Pytorch 2.0.1
- Datasets 2.12.0
- Tokenizers 0.13.2