Thanks for uploading the logs. As suspected you get a  CUDA out of memory which seems to be very close. You can find multiple of them when you search for “CUDA out of memory” in the log.
[1,3]<stdout>:RuntimeError: CUDA out of memory. Tried to allocate 52.00 MiB (GPU 3; 15.78 GiB total capacity; 14.40 GiB already allocated; 49.75 MiB free; 14.47 GiB reserved in total by PyTorch)
[1,8]<stdout>:Traceback (most recent call last):
[1,8]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,8]<stdout>:    "__main__", mod_spec)
[1,8]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,8]<stdout>:    exec(code, run_globals)
[1,8]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,8]<stdout>:    main()
[1,8]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,8]<stdout>:    run_command_line(args)

You could try using the bigger p3 instance ml.p3dn.24xlarge or maybe just try to decrease the eval_batch_size to 1. From the first log, you showed it seems training is started. Maybe you get the OOM error during eval.