Yeah you are correct, I just found out I was not implementing DDP correctly. I knew there was extra steps I needed to do to get DDP working, so was hoping that DP would speed things up, but with the large model that doesn’t seem to be the case.
I found this example 21 in fastai which uses the distributed context, so I am working on my script to add in the correct functionality. I will review the links your provided as well, it seems I need to get into the docs a little more. I appreciate your help!