In pytorch, if you apply same variable backward twice or more, it’s necessary to set the retrain_varaibles=True. Regarding your problem, I think when update the parameters with optimizer.step(), you should re-compute the loss

import torch
import torch.nn as nn
from torch.autograd import Variable


dtype = torch.FloatTensor
x = Variable(torch.randn(1, 25).type(dtype), requires_grad = True)
t = Variable(torch.randn(1, 25).type(dtype), requires_grad = False)


criterion = nn.MSELoss()
optimizer = torch.optim.Adam([x])


for i in range(5):
optimizer.zero_grad()
loss = criterion(x, t)
loss.backward()
optimizer.step()