I’m not familiar with this particular method, but if you call batch_codes.detach() while adding the noise tensor, self.embed won’t get any gradients and will thus not be updated.
Does this fit the use case?