You mean those two entries was not there during the 1 epoch, and only appears after that?

Yes, that’s correct.

What was executed after the first epoch?

After each epoch of training, evaluation is run on GPU 0.

How did you initialize DDP?

With torch.distributed.launch --nproc_per_node=2 and the following:
torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')
model.cuda(args.local_rank)
model =  torch.nn.parallel.DistributedDataParallel(model, 
                                                   device_ids=[args.local_rank],
                                                   output_device=args.local_rank,
                                                   find_unused_parameters=True)

(args.local_rank is set automatically by torch.distributed.launch)
Appreciate the response!