Validity of regression diagnostics for deterministic computer experiments The question
Is it possible to justify the use of regression diagnostics such as F-tests or the AIC for the analysis of deterministic computer experiments?
Background
One way to analyse computer experiments is by building surrogate models. The unknown function $f$ is approximated by a nice and explicitly known function $\hat f$, determined from some training data $(x_i)$ and the according responses $(f(x_i))$.
To make the question more concrete it suffices to focus on multivariate polynomials as surrogate functions. They are fit using standard regression tools (e.g. "lm" in R or "sklearn.linear_model.LinearRegression" in Python). Furthermore, since design matrices can get large very quick, methods relying on variable selection such as stepwise regression are often used.
Notice that in most cases neither of the two main regression assumptions are fulfilled: More often than not the unknown function $f$ will be ill-specified, i.e. not be a polynomial of any degree, and there will be no noisy observations, i.e. independent errors with a normal distribution.
My issue
The minimisation of residual sum of squares makes perfect sense for an approximation task. Hence, I do understand the use of regression tools in general and diagnostics such as $R^2$ even though the regression assumptions are violated.
But it seems to me that diagnostics such as F-tests and the AIC do rely crucially on the likelihood or other statistical assumptions. While tools calculate them readily (but quite formally it seems), I do not see any reason why stepwise regression or variable selection based on such diagnostics should work in the deterministic and ill-specified case.
That said, I have seen this approach used in practice and it is also mentioned in textbooks (e.g. Chapter 7.2.2 of [1]). So maybe there is something to it?
In other words: Can stepwise regression or variable selection using AIC (or similar statistics based on the likelihood or distributional assumptions of the regression model) be justified also in the case of deterministic computer experiments?
[1] "The Design and Analysis of Computer Experiments" by Thomas J. Santner, Brian J. Williams, William I. Notz

Statistical methods applied to deterministic phenomena: Statistical methods like regression analysis can be quite useful in situations where one is dealing with deterministic values, particularly in cases where the deterministic behaviour is complex enough that it is not helpful to express this behaviour in deterministic terms.  In such cases, statistical methods can be employed, where we model deterministic data using a statistical model that includes an "error term" with a particular distribution.  As with other applications, you can still use diagnostic tests to check if the (deterministic) residuals from the model appear to follow the assumed distribution, etc.  Once it is determined that a statistical model is applicable to a set of data (deterministic or not) and diagnostic tests confirm plausibility of the model assumptions, the entire suite of tests and methods applicable to that model are okay to implement.  In a regression context, this includes methods like F-tests, AIC computations, etc.
In regard to this issue, it is worth noting that simulations used in statistical practice are generated by pseudo-random number generators (PRNGs) that are deterministic in nature.  (So long as you set the seed for these generators in a deterministic manner, the resulting series of numbers is deterministic.)  Indeed, the entire field of constructing valid PRNGs and the entire field of simulation analysis using their values involves the application of statistical models and tests to deterministic phenomena.
An applied example: As an example of a regression analysis of this kind, you can have a look at O'Neill (2020) (pp. 6-9, 11-12).  That paper examines the classical occupancy distribution and computes the accuracy of the normal approximation to that distribution over a large range of parameter values.  Part of the analysis involves showing that the approximation tends to get more accurate as the parameters get large, and this is done by computing some summary quantities relating to the approximation error, and showing how they change as the parameter values increase.  To this end, the paper includes a regression analysis looking at how these summary quantities for the approximation error reduce as the parameter values become large, using a model that has extremely high goodness-of-fit.  It turns out that these quantities follow very closely (but not exactly) to a simple deterministic function and so the regression analysis is able to show that this simple relationship has a high goodness-of-fit.  What is notable in this analysis is that the data used for the regression is purely deterministive --- it consists of computed RMSE values comparing two known distributions over a set of known parameter values.  There is no randomness or real-world data in that analysis.
Philosophical considerations: When conducting this kind of analysis it is worth noting that it is possible to take an underlying philosophical position with respect to probability that does not assume the existence of (aleatory) randomness and so does not contradict determinism.  Epistemic interpretations of probability generally consider probability distributions to describe our own uncertainty in values, even if those values are fixed.  Statisticians who take an underlying epistemic interpretation of probability (which would include most Bayesians at a minimum) usually have no in-principle aversion to applying statistical methods to deterministic phenomena.
