@sshleifer
I tried dropping n_train=-1 and used 1 GPU:
python3 finetune.py \
--data_dir $NQOPEN_DIR \
--model_name_or_path t5-small --tokenizer_name t5-small \
--learning_rate=1e-4  \
--do_train \
--train_batch_size 128 \
--do_predict \
--max_source_length=12 \
--max_target_length=6 \
--eval_max_gen_length 6 \
--eval_beams=1 \
--val_check_interval 1.0 --n_val 3000 \
--output_dir $OUT --gpus 1 --logger_name wandb \
--save_top_k 3 

The result is the orange plot (fine-tuning t5-small), compared to BART outputs (in blue). As see in the figure, fine-tuning t5-small is barely moving above random predictions.
Screen Shot 2020-10-23 at 9.55.20 AM1772×1180 199 KB
Here is the content of my text_batch.json file btw:
{
    "input_ids": [
        "when does the new my hero academia movie come out",
        "when did wesley leave last of the summer wine?",
        "where is the most distortion on a robin",
        "where is each type of cartilage located in the body",
        "when were manatees put on the endangered list",
        "when was the first manmade object sent into space?",
        "who is the chief legal advisor to the government?",
        "where was the mona lisa kept during",
        "who sang the original version of true colors?",
        "who is charles off of pretty little l",
        "who won the pittsburgh steeler balt",
        "who played v in the movie v for vend",
        "who is the youngest judge currently sitting on the u",
        "who wrote variations on twinkle twinkle little star?",
        "who plays the fairy god mother in shrek 2",
        "element whose third shell contains two p electrons",
        "who played davenport in last of the summer wine",
        "i've got a brand new pair of",
        "where are the fruits of the spirit found in the bible",
        "who played mason on wizards of waverly",
        "who plays mayor hamilton on nc",
        "what is the postal code for warri nig",
        "where was sir gawain and the green",
        "the story idea for the yellow wallpaper was based on",
        "the good doctor season 1 episode 2 air date?",
        "what is the name of the college in the classic movie",
        "where is the new years eve concert held?",
        "who represented russia at the congress of vienn",
        "when did the second amendment go into effect?",
        "how much does an arleigh burke destroyer cost",
        "what cities are in santa rosa",
        "how many episodes in the itv series girlfriends?"
    ],
    "attention_mask": [
        32,
        12
    ],
    "labels": [
        "July 5, 2018",
        "2002",
        "close to the poles",
        "bronchial tubes",
        "1966",
        "3 October 1942",
        "Law Officers of the Crown",
        "the Ingres Museum",
        "Cyndi Lauper",
        "Drake",
        "Pittsburgh Steelers",
        "Hugo Weaving",
        "Neil Gorsuch",
        "Jane Taylor",
        "Jennifer Saunders",
        "Magnesium",
        "Josephine Tewson",
        "Melanie",
        "Epistle to the Galatians",
        "Sulkin",
        "Steven Robert Weber",
        "332",
        "late 14th - century",
        "rest cure",
        "October 2, 2017",
        "Faber College",
        "Westminster Central Hall",
        "Count Karl Robert Nesselrode",
        "December 15, 1791",
        "US $1.843 billion",
        "Gulf Breeze",
        "("
    ],
 "ids": [
        "",
        "s",
        "and",
        "e",
        "is",
        "i",
        "I",
        "'",
        "en",
        "your",
        "have",
        "was",
        "m",
        "!",
        "not",
        "y",
        "die",
        "A",
        "c",
        "they",
        "ul",
        "/",
        "out",
        "up",
        "if",
        "do",
        "h",
        "\u00een",
        "b",
        "other",
        "cu",
        "or"
    ],
    "decoder_input_ids": [
        "July 5, 2018",
        "2002",
        "close to the poles",
        "bronchial tubes",
        "1966",
        "3 October 1942",
        "Law Officers of the Crown",
        "the Ingres Museum",
        "Cyndi Lauper",
        "Drake",
        "Pittsburgh Steelers",
        "Hugo Weaving",
        "Neil Gorsuch",
        "Jane Taylor",
        "Jennifer Saunders",
        "Magnesium",
        "Josephine Tewson",
        "Melanie",
        "Epistle to the Galatians",
        "Sulkin",
        "Steven Robert Weber",
        "332",
        "late 14th - century",
        "rest cure",
        "October 2, 2017",
        "Faber College",
        "Westminster Central Hall",
        "Count Karl Robert Nesselrode",
        "December 15, 1791",
        "US $1.843 billion",
        "Gulf Breeze",
        "("
    ]
}

which looks normal (except the possible case of attention_mask  which I don’t quite understand).