Fast Gradient Sign Method First Order Derivative Given the infinity norm for image $x$ and its perturbed image $x\prime$ as in $R_{\infty}\left( x',x \right) =\lVert x-x' \rVert _{\infty}$, the adverserial attack:
$$
attack: x^* \gets arg\max _{x^*:R\left( x,x' \right) \le \epsilon}\mathscr{L}\left(x\prime,y \right)
$$
Then we will make a first-order assumption for the loss of the perturbed image (which is first-order Taylor expansion) that locally our loss function:
$$
\mathscr{L}\left( x',y \right) \approx \mathscr{L}\left( x,y \right) +\left( x'-x \right) ^T\nabla _x\mathscr{L}
$$
To carry out the first-order attack, all we do is substitute $\left(x' - x \right)^T\nabla _x\mathscr{L}$ with the loss function in the original equation of the attack. The equation below says that we want every element of $x$ to go in the direction of the gradient:
$$
attack:x^* \gets arg\max_{x^*:R\left(x,x' \right) \le \epsilon} \left( x'-x \right) ^T\nabla _x\mathscr{L}
$$
Question: why we did not substitute $\mathscr{L}\left( x,y \right) +\left( x'-x \right) ^T\nabla _x\mathscr{L}$ instead of $\left( x'-x \right) ^T\nabla _x\mathscr{L}$ please?

The value $\mathscr L(x,y)$ is constant with respect to $x^*$ or $x^\prime$, so optimizing $x^*$ or $x^\prime$ won't change $\mathscr L(x,y)$.
