Hi
I confirm the same issue happens also for the BERT model without any modifications, for this I run it on MRPC for 3 epochs, here is the two results:
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,542 >>   epoch                        =                3.0
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,542 >>   eval_average_metrics         = 0.8355071710663065
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,542 >>   eval_mem_cpu_alloc_delta     =                0MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   eval_mem_cpu_peaked_delta    =                1MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   eval_mem_gpu_alloc_delta     =                0MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   eval_mem_gpu_peaked_delta    =              264MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_accuracy           =             0.8088
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_combined_score     =             0.8355
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_f1                 =             0.8622
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_loss               =             0.5017
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_runtime            =         0:00:00.31
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:01,543 >>   mrpc_eval_samples_per_second =            656.083

and
[INFO|trainer_pt_utils.py:722] 2021-04-26 00:46:42,272 >> ***** test metrics *****
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   epoch                        =                3.0
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   eval_average_metrics         = 0.8656245715069244
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   eval_mem_cpu_alloc_delta     =                0MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   eval_mem_cpu_peaked_delta    =                2MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   eval_mem_gpu_alloc_delta     =                0MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   eval_mem_gpu_peaked_delta    =              264MB
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_accuracy           =             0.8431
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_combined_score     =             0.8656
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_f1                 =             0.8881
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_loss               =             0.4185
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_runtime            =         0:00:00.32
[INFO|trainer_pt_utils.py:727] 2021-04-26 00:46:42,272 >>   mrpc_eval_samples_per_second =            623.473

This now looks to me this is a library issue, @sgugger  really appreciate your comments on this issue. thanks