blade:

I have 4 GPU cards (device_count = 4). When I set the flag (indicated with the comment This is Flag ) to True , it takes 15.78 seconds to run the code. When I set it to False, it takes 0.71 seconds. Why? How could it be fixed?


One thing is that you will need a torch.cuda.synchronize() before calling time.time() to make sure all pending CUDA kernals in the stream are finished.
You can also use elapsed_time 1 to measure. See discussion here 2.
If you are looking for the most performant solution, DistributedDataParallel should be the way to go. [example 3]

When I uncomment the line  self.tools.SomeFunc(self.model, self.features)  and set the flag to  True , I receive the following error:

Looks like self.model is a DataParallel instance? If so, DataParallel does not have the first_term attribute. If this attribute is on the model instance you passed to DataParallel, you can access the original model instance through self.model.module (see DataParallel code here 3) which should have the first_term attribute.