Turns out the nn.Parameters in one of my networks weren’t getting replicated across the GPUs. I am not sure why that is. I had to include a cuda(gpu_id) on the two parameters and everything worked. No other fixes were necessary.