I am trying to implement an iterative fast gradient sign adversarial attack on a model. When I run the code for the same model, I get different accuracies for the same image. On further inspection, I found that many times for the same input the gradient differs. Is there any reason for this to happen?
runningAdv = torch.clamp( runningAdv + ((inputcopy.grad > deterministicEps).type(torch.FloatTensor) * self.perIterStep - \
                   (inputcopy.grad < -deterministicEps).type(torch.FloatTensor) * self.perIterStep).to(self.opts.gpuid) , -1 * self.epsilon, self.epsilon)
                

Initially I set deterministicEps to 0 and when this caused difference I though it may be due to precision errors as the gradients were of very small magnitude (1e-13) but even on setting deterministicEps as high as 1e-8, the output is variable.
The model is in model.eval() mode and the I generate the gradients as follows:
inp = inp.float().to(self.gpuid)
target = target.float().to(self.gpuid)
runningAdv = torch.zeros_like(inp).to(self.gpuid)
for i in range(self.niter):
                inputcopy = torch.clamp(inp + runningAdv, 0, 1)
                inputcopy.requires_grad = True
                out = self.model(inputcopy)
                loss = self.loss(out, target)
                loss.backward()
                runningAdv = torch.clamp( runningAdv + ((inputcopy.grad > deterministicEps).type(torch.FloatTensor) * self.perIterStep - \
                   (inputcopy.grad < -deterministicEps).type(torch.FloatTensor) * self.perIterStep).to(self.opts.gpuid) , -1 * self.epsilon, self.epsilon)

Why should there be changes in the gradient for the same image (even if the magnitude of the gradient is very small, shouldn’t its value remain consistent when I use the same input and model)