Hey @elistevens
Looks like you already using DDP with one device per processes, which is the recommended setup.
Can you try different OMP_NUM_THREADS configurations?  Does it speed up or slow down if you set OMP_NUM_THREADS to 1? Sometimes DataLoader can also cause slowdowns. If does it affect the performance if you get rid of the DataLoader by using synthetic generated input/output (just for testing purpose)?