@FarisHijazi Could you share which torch version are you using? I tried the DDP script on my local box using PyTorch 1.9 and it is running fine:
3276it [00:10, 309.25it/s]Epoch [2/2], Step [3300/3750], Loss: 0.4405
3371it [00:10, 309.47it/s]Epoch [2/2], Step [3400/3750], Loss: 0.2047
3468it [00:10, 314.98it/s]Epoch [2/2], Step [3500/3750], Loss: 0.3345
3596it [00:11, 308.55it/s]Epoch [2/2], Step [3600/3750], Loss: 0.3061
3691it [00:11, 309.97it/s]Epoch [2/2], Step [3700/3750], Loss: 0.2019
3750it [00:11, 317.59it/s]
Training complete in: 0:00:23.282688

Even the DataParallel script is working as expected:
net(x)
3.030186414718628
net(x)
0.0012230873107910156
net(x)
0.0010361671447753906
net(x)
0.0010747909545898438
net(x)
0.0009708404541015625
net(x)
0.0010142326354980469
net(x)
0.0009481906890869141
net(x)
0.0009622573852539062
net(x)
0.0009491443634033203
net(x)
0.0009527206420898438