Bug
I have only a functional description of the bug. Still trying to make a MWE.
Sometimes when using an Embedding in a model with DataParallel I hit errors like so:
x = ... # shape: [<batch part>, <something>]
y = self.embed(x) # shape: [<WRONG batch part>, <something>, <emb dim>]
#^ the return value of the Embedding is "incomplete" on some GPUs.
# Noticed it most on cuda:1, but that shouldn't matter.
# E.g. <batch> = 2000, <wrong batch> = 11

To Reproduce
Steps to reproduce the behavior:

Have a model with Embedding.
Use DataParallel on the model such that you’re close to saturating your system.
Things shouldn’t end up working, if the bug is reproducible.
You don’t need anything fancy. Just some pretrained model in eval mode and pass it some input.


Expected behavior
DataParallel models behave functionally identically to normal models. Convergence and gradient descent questions notwithstanding.
Environment

PyTorch Version (e.g., 1.0): 1.5 ~ 1.6.x-dev
OS (e.g., Linux): Ubuntu 16.04 LTS
How you installed PyTorch (conda, pip, source): conda
Build command you used (if compiling from source): N/A
Python version: 3.7.7
CUDA/cuDNN version: 10.2
GPU models and configuration: Geforce GTX Titan X * 2
Any other relevant information: N/A