Honestly, I would recommend starting from a simpler version and gradually adding complexities to it.

Okay, I just did the above using the example you provided. It is working now, even when I use backend='nccl' too.
But here is an issue,  how to merge the metrics provided by different GPUs together. Now, I am getting 3 outputs like below -


Epoch: 1 	Training Loss: 1.044399 	Validation Loss: 21.517847

Test Accuracy: 58% (106/180)
Validation loss decreased (inf --> 21.517847).  Saving model ...
Epoch: 1 	Training Loss: 1.036141 	Validation Loss: 23.852943

Test Accuracy: 47% (86/180)
Epoch: 1 	Training Loss: 1.040907 	Validation Loss: 21.792797

Test Accuracy: 56% (101/180)
Validation loss decreased (inf --> 23.852943).  Saving model ...
Validation loss decreased (inf --> 21.792797).  Saving model ...
Epoch: 2 	Training Loss: 0.971241 	Validation Loss: 19.095814

Test Accuracy: 62% (113/180)
Validation loss decreased (inf --> 19.095814).  Saving model ...

wandb: Waiting for W&B process to finish, PID 30011
Epoch: 2 	Training Loss: 0.965157 	Validation Loss: 20.114439

Test Accuracy: 55% (99/180)
Epoch: 2 	Training Loss: 0.966274 	Validation Loss: 21.474331

Test Accuracy: 56% (101/180)
Validation loss decreased (inf --> 20.114439).  Saving model ...
Validation loss decreased (inf --> 21.474331).  Saving model ...