Hey ptrblck,
I found a interesting bug in my code. I have a tensor namely carrier_seqlen , and the size equals with batch-size, e.g., when the batch-size equals 10, the size of carrier_seqlen  should be 10 too.
Here is the issue: when I print out the tensor carrier_seqlen both inside and outside my model encoder-decoder, the value on device 0 changes:

carrier_seqlen outside encoder-decoder:  tensor([504, 385, 332, 324, 271, 271, 260, 245, 238, 169], device=‘cuda:0’,
dtype=torch.int32) torch.Size([10]) cuda:0 torch.int32


carrier_seqlen in encoder-decoder:  tensor([257, 257, 257, 257, 257], device=‘cuda:0’, dtype=torch.int32) torch.Size([5]) cuda:0 torch.int32
carrier_seqlen in encoder-decoder:  tensor([271, 260, 245, 238, 169], device=‘cuda:1’, dtype=torch.int32) torch.Size([5]) cuda:1 torch.int32

I am not sure why this happened? I just pass this tensor inside the model and haven’t done any processing, the size is torch.Size([10]), and it should be divided into  torch.Size([5]) cause I have two gpus inside the model, right? But why the value changed?
Thanks for your help!