Has anyone had RuntimeError: all tensors must be on device[0] errors inside a forward pass?
For example, I have this in my training loop:
mixed, clean = batch[0], batch[1]
znoise = znoise.resize_(batch_size, 1024, 8).normal_(0., 1).float()
noise_clean = pnoise.resize_(clean.size()).normal_(0, 1) * input_noise_std
mixed = to_gpu(mixed, inference=False, fp16=fp16)
clean = to_gpu(clean, inference=False, fp16=fp16)
znoise = to_gpu(znoise, inference=False, fp16=fp16)
pnoise = to_gpu(pnoise, inference=False, fp16=fp16)
print("mixed", mixed.get_device()) # outputs 0
print("znoise", znoise.get_device()) # outputs 0

model(mixed, znoise) with forward:
def forward(self, wav, znoise):
    print(wav.get_device(), znoise.get_device(), cuda.current_device()) # outputs 0, 0, 0 then 1,1,1
    # encoder-decoder ladder network with wav and znoise
    # ...
    return output_dec

Executing model(mixed, znoise) leads to RuntimeError: all tensors must be on device[0]