Also, res101 should be set to training mode right? I.e., res101.training().
Otherwise, if you set input1 = Variable(torch.zeros(1,3,224,224),requires_grad=True) and same for input2 on every iteration then it should work…