If you install from source, you can edit finetune.py to do as you describe 
The function you want is here 14, and you could introduce a switch to only generate if called from test_step.
Validation loss is also calculated within this function and is the basis upon which checkpoints are ranked (as far as I understand)