Where do you create this tensor? Inside the wrapped model’s forward? Then assuming that x is the input to the model then you should use x.new_tensor() or x.new_zeros() or torch.zeros_like() to create the tensor automatically in the correct device.
See here: https://pytorch.org/docs/stable/tensors.html#torch.Tensor.new_zeros 19
If you are assigning the attribute from the outside e.g model.tensor = some_tensor then this is not going to work. The model is replicated to each device after each iteration. Doing model.tensor does not assign the attribute to the real model inside the wrapper but directly to the DataParallel instance.