readme: # Roberta2Roberta Summarization with 🤗 EncoderDecoder Framework
  
  This model is a Roberta2Roberta model fine-tuned on summarization.
  
  Roberta2Roberta is a `EncoderDecoderModel`, meaning that both the encoder and the decoder are `roberta-base` 
  RoBERTa models. Leveraging the [EncoderDecoderFramework](https://huggingface.co/transformers/model_doc/encoderdecoder.html#encoder-decoder-models), the 
  two pretrained models can simply be loaded into the framework via:
  
  ```python
  roberta2roberta = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base")
  ```
  
  The decoder of an `EncoderDecoder` model needs cross-attention layers and usually makes use of causal 
  masking for auto-regressiv generation.
  Thus, ``roberta2roberta`` is consequently fined-tuned on the `CNN/Daily Mail`dataset and the resulting model
  `roberta2roberta-cnn_dailymail-fp16` is uploaded here.
  
  ## Example
  
  The model is by no means a state-of-the-art model, but nevertheless 
  produces reasonable summarization results. It was mainly fine-tuned 
  as a proof-of-concept for the 🤗 EncoderDecoder Framework.
  
  The model can be used as follows:
  
  ```python
  from transformers import BertTokenizer, EncoderDecoderModel
  
  model = EncoderDecoderModel.from_pretrained("patrickvonplaten/roberta2roberta-cnn_dailymail-fp16")
  tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
  
  article = """(CNN)Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members singing a racist chant. SAE's national chapter suspended the students, but University of Oklahoma President David B
  oren took it a step further, saying the university's affiliation with the fraternity is permanently done. The news is shocking, but it's not the first time SAE has faced controversy. SAE was founded March 9, 185
  6, at the University of Alabama, five years before the American Civil War, according to the fraternity website. When the war began, the group had fewer than 400 members, of which "369 went to war for the Confede
  rate States and seven for the Union Army," the website says. The fraternity now boasts more than 200,000 living alumni, along with about 15,000 undergraduates populating 219 chapters and 20 "colonies" seeking fu
  ll membership at universities. SAE has had to work hard to change recently after a string of member deaths, many blamed on the hazing of new recruits, SAE national President Bradley Cohen wrote in a message on t
  he fraternity's website. The fraternity's website lists more than 130 chapters cited or suspended for "health and safety incidents" since 2010. At least 30 of the incidents involved hazing, and dozens more invol
  ved alcohol. However, the list is missing numerous incidents from recent months. Among them, according to various media outlets: Yale University banned the SAEs from campus activities last month after members al
  legedly tried to interfere with a sexual misconduct investigation connected to an initiation rite. Stanford University in December suspended SAE housing privileges after finding sorority members attending a frat
  ernity function were subjected to graphic sexual content. And Johns Hopkins University in November suspended the fraternity for underage drinking. "The media has labeled us as the 'nation's deadliest fraternity,
  ' " Cohen said. In 2011, for example, a student died while being coerced into excessive alcohol consumption, according to a lawsuit. SAE's previous insurer dumped the fraternity. "As a result, we are paying Lloy
  d's of London the highest insurance rates in the Greek-letter world," Cohen said. Universities have turned down SAE's attempts to open new chapters, and the fraternity had to close 12 in 18 months over hazing in
  cidents."""
  
  input_ids = tokenizer(article, return_tensors="pt").input_ids
  output_ids = model.generate(input_ids)
  
  print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
  # should produce
  # Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members singing racist chants. The fraternity's national chapter has had to close 12 in 18 months over hazing. 
  # Sigma has had more than 130 chapters in 18 states. University of Oklahoma president says fraternity has been "deteriorated".
  ```
  
  ## Training script:
  
  **IMPORTANT**: In order for this code to work, make sure you checkout to the branch 
  [more_general_trainer_metric](https://github.com/huggingface/transformers/tree/more_general_trainer_metric), which slightly adapts 
  the `Trainer` for `EncoderDecoderModels` according to this PR: https://github.com/huggingface/transformers/pull/5840. 
  
  The following code shows the complete training script that was used to fine-tune `roberta2roberta-cnn_dailymail-fp16
  ` for reproducability. The training last ~9h on a standard GPU.
  
  ```python
  #!/usr/bin/env python3
  import nlp
  import logging
  from transformers import RobertaTokenizer, EncoderDecoderModel, Trainer, TrainingArguments
  
  logging.basicConfig(level=logging.INFO)
  
  model = EncoderDecoderModel.from_encoder_deco