It depends whether you want gradients to backpropagate through the update.
It looks like you’re doing a masked gradient step, so these would typically not be backpropagated through.
In this case: Just wrap the two statements in with torch.no_grad():. You could try if “+=” and inplace clamp_ is faster.
If you want to propagate through to the previous version of perturbed, you would need to do the masking first:
update = torch.zeros_like(perturbed)  # doesn't require grad
update[masks] = (gradient * epsilon)[masks] # because update didn't require grad, this works with autograd
perturbed = (perturbed + update).clamp(0, 255) # we replace the name perturbed with a new version, so no inplace business

Best regards
Thomas