Is $R^2$ or RMSE more important for determining the success of a neural network regression? I have designed a neural network with two different choices of features.
Neither feature set stands out as one trains a neural network that scores higher $R^2$ coefficient on test data whereas the other feature set produces a neural network that scores the lower RMSE.
Which of $R^2$ and RMSE should be the deciding factor for which feature set is better?

They are equivalent up to computational issues that arise from doing math on a computer.
$$R^2=1-\dfrac{SSResiduals}{SSTotal}$$
$$
RMSE = \sqrt{\dfrac{SSResiduals}{n}}
$$
Both are just functions of the sum of the squares residuals (“errors” in a mostly-acceptable-even-if-technically-incorrect machine learning slang).
Setting aside some issues that can arise from doing math on a computer, maximizing $R^2$ is the same as minimizing $RMSE$.
