pytorch-newbie:

retain_graph=True


I think, retain_graph=True is unnecessary.
I feel bad to write spaghetti code. But try ths 
syn_energy.sum().backward()

# creating next input
new_sample_seq = sample_seq.detach()
temp = new_sample_seq + 0.5 * self._opt.sampling_step * self._opt.sampling_step * sample_seq.grad
new_sample_seq = new_sample_seq * (1-mask) + temp * mask
new_sample_seq.clamp_(0.0, 1.0) # min , max
new_sample_seq.requires_grad = True
sample_seq = new_sample_seq

Basically, I am detaching the input variable so that it doesn’t get any gradients from these operations.