Yup, just like the code you posted. So feature_target and feature_identity are fixed in the loop.
Just to understand more about retain_graph, why should I retain it? Which information from (i-1)-th iteration does i-th iteration needs. As far as I know, there should be none, feature_target and feature_identity should be treated as constants. What am I missing here?
feature_target = model(input1)
feature_identity = model(input2)
feature_adversarial = model(data)
for i in range(100):
    distance_target = torch.sum((feature_adversarial - feature_target.clone()) **2)
    distance_identity = torch.sum((feature_adversarial - feature_identity.clone()) ** 2)
    loss = distance_target + 2*distance_identity
    loss.backward()
    feature_adversarial = update(feature_adversarial)
    model(feature_adversarial)

My code is something more like this.