RMSE vs MSE loss function - the optimization solutions are equivalent? If we optimize a function $f$ with respect to loss $L$, which is defined as RMSE; Are we going to get the same solution as optimizing MSE ? Even, if the function $f$ is non-linear (e.g. a neural network) ?

In addition to @Tim's (+1) answer, I'd like to add some other perspective (also repeating some things). If some $x^*$ are the minimizers of RMSE ($\geq 0$), they're the minimizer of MSE, because the operation is monotonic, e.g. ($2 < 3 \rightarrow 2^2<3^2$). So, the same set of global optimizers, if there exists more than one, exist for the MSE.
But, optimization algorithms aren't perfect, and end up in local optima. A slight change in loss function will affect your gradient steps (e.g. the corresponding learning rates for the two loss functions will be different), therefore may lead you to another local optimum. So, it's quite possible to end up in different locations.
