I use a self-attention layer in my network but Unfortunately I got cuda out of memory error even though I reduced the batch_size to one only
So, I tried to run the code on two GPUs but I got this error
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)


I moved the model and the input image to cuda0 to be on similar gpu but each time I move one item to cuda0 I found a new item should be on the same gpu too 
how could I resolve this problem please? any help