I’m using the example command line supplied with run_mlm.py (v4.4.1), but I’ve enabled more frequent logging as well as evaluation at the end of each epoch:
python run_mlm.py --model_name_or_path roberta-base \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--output_dir test-wikitest \
--do_train \
--do_eval \
--evaluation_strategy epoch \
--logging_steps 10 

By default, this is configured to train over 3 epochs.  When I run this to completion, I find that the eval_loss calculated at the end of the last epoch is reported as:
[INFO|trainer.py:1775] 2021-03-18 18:16:19,116 >> ***** Running Evaluation *****
[INFO|trainer.py:1776] 2021-03-18 18:16:19,116 >>   Num examples = 496
[INFO|trainer.py:1777] 2021-03-18 18:16:19,116 >>   Batch size = 8
{'eval_loss': 1.2713477611541748, 'eval_runtime': 27.0394, 'eval_samples_per_second': 18.344, 'epoch': 3.0}

Immediately following this, the final evaluation (because I specified --do_eval) is executed.  But the eval_loss reported is different:
[INFO|trainer.py:1775] 2021-03-18 18:16:47,957 >> ***** Running Evaluation *****
[INFO|trainer.py:1776] 2021-03-18 18:16:47,957 >>   Num examples = 496
[INFO|trainer.py:1777] 2021-03-18 18:16:47,957 >>   Batch size = 8
[INFO|trainer_pt_utils.py:650] 2021-03-18 18:17:15,018 >> ***** eval metrics *****
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   epoch                     =     3.0
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_loss                 =  1.2644
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_mem_cpu_alloc_delta  =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_mem_cpu_peaked_delta =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_mem_gpu_alloc_delta  =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_mem_gpu_peaked_delta =  1584MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_runtime              = 26.8687
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_samples              =     496
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,018 >>   eval_samples_per_second   =   18.46
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:17:15,019 >>   perplexity                =  3.5409

I was expecting the two eval_loss values to be the same, since no training happens between the two back-to-back evaluations and since the validation set is the same.
Further, when I invoke run_mlm.py with the trained model checkpoint and do only an evaluation (no training at all):
python run_mlm.py --model_name_or_path test-wikitest \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--output_dir test-wikitest-evalonly \
--do_eval 

I get still another value for eval_loss:
[INFO|trainer.py:1775] 2021-03-18 18:46:12,713 >> ***** Running Evaluation *****
[INFO|trainer.py:1776] 2021-03-18 18:46:12,713 >>   Num examples = 496
[INFO|trainer.py:1777] 2021-03-18 18:46:12,714 >>   Batch size = 8
[INFO|trainer_pt_utils.py:650] 2021-03-18 18:46:35,544 >> ***** eval metrics *****
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_loss                 =   1.287
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_mem_cpu_alloc_delta  =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_mem_cpu_peaked_delta =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_mem_gpu_alloc_delta  =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_mem_gpu_peaked_delta =  1584MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_runtime              = 22.6793
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_samples              =     496
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   eval_samples_per_second   =   21.87
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   init_mem_cpu_alloc_delta  =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   init_mem_cpu_peaked_delta =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   init_mem_gpu_alloc_delta  =   476MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   init_mem_gpu_peaked_delta =     0MB
[INFO|trainer_pt_utils.py:655] 2021-03-18 18:46:35,545 >>   perplexity                =  3.6218

Can anyone tell me what the source of the non-determinism is?
Thanks in advance!