I have found the problem. The exact problem is the tensor type. We must make sure what is our tensor input and what is exactly our network wants. You can print the tensor first to see the tensor type. The most big mistake is when I am using torch.from_numpy which return DoubleTensor from my numpy type.