Validity of regression diagnostics for deterministic computer experiments The question
Is it possible to justify the use of regression diagnostics such as F-tests or the AIC for the analysis of deterministic computer experiments?
Background
One way to analyse computer experiments is by building surrogate models. The unknown function $f$ is approximated by a nice and explicitly known function $\hat f$, determined from some training data $(x_i)$ and the according responses $(f(x_i))$.
To make the question more concrete it suffices to focus on multivariate polynomials as surrogate functions. They are fit using standard regression tools (e.g. "lm" in R or "sklearn.linear_model.LinearRegression" in Python). Furthermore, since design matrices can get large very quick, methods relying on variable selection such as stepwise regression are often used.
Notice that in most cases neither of the two main regression assumptions are fulfilled: More often than not the unknown function $f$ will be ill-specified, i.e. not be a polynomial of any degree, and there will be no noisy observations, i.e. independent errors with a normal distribution.
My issue
The minimisation of residual sum of squares makes perfect sense for an approximation task. Hence, I do understand the use of regression tools in general and diagnostics such as $R^2$ even though the regression assumptions are violated.
But it seems to me that diagnostics such as F-tests and the AIC do rely crucially on the likelihood or other statistical assumptions. While tools calculate them readily (but quite formally it seems), I do not see any reason why stepwise regression or variable selection based on such diagnostics should work in the deterministic and ill-specified case.
That said, I have seen this approach used in practice and it is also mentioned in textbooks (e.g. Chapter 7.2.2 of [1]). So maybe there is something to it?
In other words: Can stepwise regression or variable selection using AIC (or similar statistics based on the likelihood or distributional assumptions of the regression model) be justified also in the case of deterministic computer experiments?
[1] "The Design and Analysis of Computer Experiments" by Thomas J. Santner, Brian J. Williams, William I. Notz

It sounds like your deterministic experiments are effectively evaluating an unknown function $f$ at many different input values $x$. If evaluating $f$ at new $x$ values is cheap, then you can easily get arbitrarily large samples; so hypothesis tests (which are partly a measure of sample size) aren't going to be a very useful way to choose a parsimonious $\hat f$. Similar arguments can be made for AIC.
Instead, I'd suggest you think of a measure of "How good does my approximation need to be to be useful for my purposes?" What are the units of $f$? Choose the smallest value of a maximum or average discrepancy between $f$ and $\hat f$ that would be acceptable for your needs, on the range of $x$ values that you care about. Then, evaluate $f$ at many $x$ values, so that you have more than enough data to get low variance in the coefficient estimates when fitting fairly complex $\hat f$ models. Choose the simplest of these $\hat f$ models that meets your criteria for a "good enough" approximation.

On the other hand, if evaluating $f$ for new $x$ values is expensive or slow and your sample size is essentially fixed, then the F test or AIC might help you choose the most-complex $\hat f$ you can afford to fit with this sample size. The fact that $f$ is deterministic isn't a problem (though it might affect your choice of which test to carry out; see below).
It often helps to think of significance tests (like the F test) as a measure of sample size. I will assume you're using the F test that compares two nested regression models. Then the F test is basically asking: "Is my sample large enough to trust the larger / more complex model? Or is there too much variability relative to this sample size, and it would be safer to use the smaller / less complex model at this sample size?"
Say the true unknown $f$ is pretty close to a quadratic function (but not quite exactly quadratic). And say you first try to compare a linear $\hat {f_1}$ vs a quadratic $\hat {f_2}$ fit to the pairs $(x, f(x))$. If you've evaluated $f$ at very few values of $x$, the F test might fail to reject $H_0$, telling you that you don't have enough data and your estimated $\hat {f_2}$ is too noisy. But if you've evaluated $f$ at enough different $x$ values, the F test might reject $H_0$, telling you that you've got enough data to safely trust your estimated $\hat {f_2}$ as a better approximation than $\hat {f_1}$.
But this issue of "Is my sample big enough?" is distinct from "Is $\hat f$ a good enough approximation to $f$?" The latter question requires subject matter expertise. Again, imagine that $f$ is nearly-but-not-quite quadratic. And let's say that the true differences between $f$ and its best quadratic approximation are negligible for your purposes; a cubic $\hat{f_3}$ could technically be an even better fit, but it would make almost no practical difference in whatever you're using $\hat f$ for. Even so, if you have a large enough sample size, the F test comparing quadratic vs cubic will reject $H_0$ and say you've got enough data to trust the cubic approximation $\hat{f_3}$.
My key point is that "negligible for your purposes" is something that you must decide and the F test cannot answer for you. I don't know the context of your computer experiments, but here's a physics example. Say you're trying to predict $f$ = boiling point of water from $x$ = atmospheric pressure. The true relationship is not linear, but if your purpose is to suggest approximate baking times in a recipe book, a linear $\hat f$ might be plenty good enough. On the other hand if your purpose is to design high-performance meteorological equipment, a linear $\hat f$ may be nowhere near good enough and you'll need a more complex model (and enough data to fit it).

Finally, if you do use hypothesis tests in this scenario, and your deterministic $f$ is smooth and your $x$ values are very close together... then the independent-residuals assumption isn't going to be appropriate.
Think of time-series data. Imagine an extreme example: $y$ is changing once a day (and is independent from day to day), but I record measurements every hour. Then I don't really have $n$ independent measurements -- my "effective sample size" is more like $n/24$. If I fit a regression model to $y$ vs $time$ and don't account for this autocorrelation, my F tests will wrongly think that I do have $n$ independent measurements, and my p-values will be way smaller than they really ought to be for the effective sample size.
In your case it might not be quite as extreme. But I'd still look into autocorrelation models to account for the fact that similar $x$ values might have extremely similar $f$ values.
