For those who are facing similar memory issue, look at the autograd setting: Volatile.
It’s recommended for inference mode, to optimize the amount of memory used in evaluating the model .
http://pytorch.org/docs/master/notes/autograd.html#volatile 869
you could do sth like this:
volatile_input = Variable(torch.randn(1000,1000000), volatile=True).cuda()