I figured it out. Made a silly mistake. I was doing embedding lookups on both GPUs when the embeddings were obviously only on one. I was foolishly moving the embeddings between GPUs for the forward pass, but obviously, the backward pass was not doing this on my behalf, and therefore blowing up.