Btw, I tried BART with the same script and it seems to be working much better:
export NQOPEN_DIR=/home/danielk/nqopen_csv
export OUT=/home/danielk/fine_tune_bart_oct21 

python3 finetune.py \
--data_dir $NQOPEN_DIR \
--model_name_or_path facebook/bart-base --tokenizer_name facebook/bart-base \
--learning_rate=3e-4 --freeze_encoder --freeze_embeds \
--do_train --train_batch_size 16 \
--do_predict --n_train -1 \
--val_check_interval 0.25 --n_val 3000 \
--output_dir $OUT --gpus 4 --logger_name wandb \
--save_top_k 3 

The following ROUGE graph shows that BART-base (blue)  is doing much better T5-small (brown). T5-small (brown) barely gets off the ground which is quite surprising.
Screen Shot 2020-10-21 at 5.40.58 PM1358×1250 93.9 KB