I just added two notebooks showing how to reproduce the results in the paper.
Here one for Bert2Bert on CNN/Dailymail:

  
      github.com
  
  
    patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb 67
{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT2BERT for CNN/Dailymail",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMYchE5beUGXDKxJoG2JsLp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "e13b8fdb8a564e8699c77890e29995bf": {
          "model_module": "@jupyter-widgets/controls",


  This file has been truncated. show original

  
  
    
    
  
  


Here one for RobertaShared on BBC XSum:

  
      github.com
  
  
    patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb 23
{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RoBERTaShared for BBC XSum",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNjqhK5uRHETrMPoFcaBl9P",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "675630bf7b35483f9a424eb7433ab76a": {


  This file has been truncated. show original

  
  
    
    
  
  


The Bert2Bert model actually performs a bit better than reported in the paper, the roberta_shared model a bit worse (but training roberta_shared a bit longer would probably close that gap).
The motivation of doing this is to provide some educational material on how to use the EncoderDecoderModel - the exact performance was less important here.
I’m planning on making two short notebooks on Roberta2GPT2 for sentence fusion (DiscoFuse) and a Bert2Rnd for WMT en-de. Hope this is useful!