mrshenli:

See the data parallel implementation below:


I didn’t quite get your point on this one.



 mrshenli:

My suspicion is that the model parameters are so light that the overhead of GIL contention, input scattering, output gathering, and model replication in DataParallel forward pass overshadows the speedup brought by multi-gpu training.


That makes sense. Let me try it on a more sophisticated code and update the answer.



 mrshenli:

Are these parameters used in real applications or are you trying to profile DataParallel performance?


I’m just making sure I understand how to use DataParallel correctly. Although way simpler, this sample code mimics the general structure of my actual code. So, there is nothing wrong with my implementation?



 mrshenli:

you can access the original model instance through self.model.module (see DataParallel code here ) which should have the first_term attribute.


Thanks, based on your response I used this wrapper to get access to attributes without altering the code itself.