Thanks for the update! The profiling looks better now. 



 keltis:

Since the batches contain data of different lengths, could this be the origin of the problem?


That could be a possible reason.
However, we generally recommend to use DistributedDataParallel with a single process per GPU to avoid the scatter and gather of the model in each iteration, which might add some overhead.