@Anakin @abhidipbhattacharyya I have met a similar issue, when implementing multi-head attention, I need to use the function troch.masked_fill(), and the code runs fine on single gpu but got an dimension mismatch error on multi-gpus. In my situation, I found that the first dimension of the mask I pass to the forward method is not batch_size, but DataParallel will split the tensor on dimension 0 by default, causing the mismatch problem. So what we need to do is adding a batch_size dimention to the mask, using mask=mask.unsqueeze(0) before you pass the mask to the model, and after split, all device will get the same mask because the batch_size is 1 here. Also you will need to remember your mask already has a batch_size dimension and don’t need to add this dimension in self-attention logic.
To sum up, ensure all tensors you pass to the DataParallel module have a batch_size dimension.