@valhalla thanks for the feedback!
It doesn’t seem like there is any --length_penalty parameter for finetune.py, but I think set the rest of the parameters you suggested. I repeated the experiment with the following commands:
export NQOPEN_DIR=/home/danielk/nqopen_csv
export OUT=/home/danielk/fine_tune_t5_small_9

python3 finetune.py \
--data_dir $NQOPEN_DIR \
--model_name_or_path t5-small --tokenizer_name t5-small \
--learning_rate=1e-4 --freeze_encoder --freeze_embeds \
--do_train --train_batch_size 16 \
--do_predict --n_train -1 \
--max_source_length=32 \
--max_target_length=16 \
--eval_max_gen_length 16 \
--eval_beams=1 \
--val_check_interval 0.25 --n_val 3000 \
--output_dir $OUT --gpus 4 --logger_name wandb \
--save_top_k 3 

Here is the output (pink) and unfortunately it’s no bueno. ROUGE==5% is basically barely above random output.
Screen Shot 2020-10-22 at 12.57.08 PM2686×1272 288 KB
If the fine-tuning works, it should quickly move north of ROUGE==25%.