What is your objective function for validation loss? I think the answer to your question depends on several factors, the biggest of which are: the kind of rouge you are using (R1 vs. R2 vs. RL) and the score range (high vs. low).