Validity of regression diagnostics for deterministic computer experiments The question
Is it possible to justify the use of regression diagnostics such as F-tests or the AIC for the analysis of deterministic computer experiments?
Background
One way to analyse computer experiments is by building surrogate models. The unknown function $f$ is approximated by a nice and explicitly known function $\hat f$, determined from some training data $(x_i)$ and the according responses $(f(x_i))$.
To make the question more concrete it suffices to focus on multivariate polynomials as surrogate functions. They are fit using standard regression tools (e.g. "lm" in R or "sklearn.linear_model.LinearRegression" in Python). Furthermore, since design matrices can get large very quick, methods relying on variable selection such as stepwise regression are often used.
Notice that in most cases neither of the two main regression assumptions are fulfilled: More often than not the unknown function $f$ will be ill-specified, i.e. not be a polynomial of any degree, and there will be no noisy observations, i.e. independent errors with a normal distribution.
My issue
The minimisation of residual sum of squares makes perfect sense for an approximation task. Hence, I do understand the use of regression tools in general and diagnostics such as $R^2$ even though the regression assumptions are violated.
But it seems to me that diagnostics such as F-tests and the AIC do rely crucially on the likelihood or other statistical assumptions. While tools calculate them readily (but quite formally it seems), I do not see any reason why stepwise regression or variable selection based on such diagnostics should work in the deterministic and ill-specified case.
That said, I have seen this approach used in practice and it is also mentioned in textbooks (e.g. Chapter 7.2.2 of [1]). So maybe there is something to it?
In other words: Can stepwise regression or variable selection using AIC (or similar statistics based on the likelihood or distributional assumptions of the regression model) be justified also in the case of deterministic computer experiments?
[1] "The Design and Analysis of Computer Experiments" by Thomas J. Santner, Brian J. Williams, William I. Notz


In other words: Can stepwise regression or variable selection using AIC (or similar statistics based on the likelihood or distributional assumptions of the regression model) be justified also in the case of deterministic computer experiments?

Here's an argument for likelihood-based statistics. Let $Y = f(X)$, where $f$ is a deterministic function. Assume you observe a finite number of realizations s.t. $\mathbf{y} = f(\mathbf{X})$. If you knew both $f$ and $\mathbf{X}$, then you would know $\mathbf{y}$ with probability one, or very vaguely $P(\mathbf{y} = f(\mathbf{X}) | \mathbf{X}, f) = 1$. You don't know $f$, so you place a Gaussian process prior $f | \mathbf{X}\sim\mathcal{GP}(\cdot, \cdot)$. You integrate out $f$ to obtain the output marginal likelihood, which is MVN Gaussian: $\mathbf{y} | \mathbf{X}\sim\mathcal{MVN}(\cdot, \cdot)$. Even if the computer code output is deterministic and there's no "innate" variability on it, the output marginal likelihood is multivariate normal due to your uncertainty on $f$. So, statistics based on this marginal likelihood seem sensible to me.
More general, here are a few things that might help you decide how comfortable you feel about using such diagnostic techniques.

*

*Your computer model is deterministic, i.e., $Y = f(X)$. Since the computer model does not have a random error component, there's no aleatoric uncertainty. However, you don't know $f$, so there's epistemic uncertainty. See Aleatoric and epistemic.

*Even though the underlying function $f$ might be deterministic, the output value is unknown a priori (before running the simulator). This is sometimes called code uncertainty and is enough to justify the use of a stochastic process.

*Surrogates are most frequently set up in a Bayesian framework, and so the probabilistic statements are about your uncertainty about the function that you are approximating $f$ and not really about the inherent random noise in your computer code. This applies even when Empirical Bayes is used, which in many contexts can get quite close to simply "find MLE estimates but interpret like a Bayesian model".

*Bastos and O'Hagan (2009, sec. 3.1) discusses the challenges associated with diagnostics for linear models fitted to a deterministic function. In particular, the training set predictions will be perfect if you use an interpolator, thus the need to use cross-validation or a new data set. Out-of-sample validation introduces some uncertainty. However, they also note that there might be scoring functions for deterministic computer experiments such as the Mahalanobis distance. See 10.1198/TECH.2009.08019
An adjacent note.

*

*A linear model might not be commonly used for a surrogate. However, you can cast a linear model as a Gaussian process with linear kernel $\mathbf{XX}^\top$ (Rasmussen and Williams, 2006, sec. 4.2.2).

References

*

*Bastos, L. S., & O’Hagan, A. (2009). Diagnostics for Gaussian process emulators. Technometrics, 51(4), 425–438. https://doi.org/10/bw62bq

*Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. MIT Press.

