It seems that in this line
x = F.relu(F.max_pool2d(self.conv2_drop(conv2_in_gpu1), 2))

conv2_in_gpu1 is still on GPU1, while self.conv2_drop etc. are on GPU0. You only transferred x back to GPU0.
Btw, what is your use case?