Oh I didn’t read that, sorry. Then what you really need to do is is to set the input Variable or any parameter in the model as volatile which makes sure to use the least amount of memory possible. This is used for inference mode since you don’t even need to call backward() for inference. You can use it like this:
input = Variable(torch.randn(1, 3, 227, 227), volatile=True) 
 model(input)

Something like this might improve your speed considerably. Although what I said before still applies.