Is it possible to evaluate a given model without having access to its fit method? I have a data set with one real-valued feature and a real-valued target. Someone has used this data set to fit a model (a regression). I get a results of this fit, which is a single function mapping from the feature to the target.
My question is if it is possible to estimated the level of over-fit (or under-fit) just by looking (in a broad sense) at this single function and the given data set?
I guess that it should be possible because by a visualization one could potentially see if the function has some systematic deviations from the data (for example data points assume a quadratic dependency and the model is just a linear function). So, in this case, we can conclude that we have an under-fit.
Similarly, in some cases we could see that the function is too sensitive to noise and do not reflect real dependency. So, in this case, we can conclude that we have an over-fit.
However, I would like to have a more rigorous (quantitative, automatic) method instead of a manual visualization. Moreover, if we have several features instead of one, a visualization becomes problematic.
I would like to emphasize again, that the only thing that I have is a model (function mapping from features to targets) and a data set that was used to find this function (so I have only in-sample results). In other words, I cannot retrain a model and, let's say, run a K-Fold cross validation.

Suppose for example your data has 5 points and the fitted function is a 5-th degree polynomial that exactly passes through all of them. Can you say just by looking at the data if this is an over-fit ?
It could be, for example, that those points represents precise measurements of some physical system (such as the trajectory of a planet) which has a theoretical model that is known to be a 5-th degree polynomial, where the coefficients are free parameters. Then 5 data points are enough to determine all the unknown parameters, and the fit is perfectly valid.
On the hand it could be of course that such a model is not known a-priory, or that the measurements have large uncertainties, and in that case the same function would be completely over-fitted and meaningless. (Conversely,  someone could fit a linear function when the model is known to be 5-th degree,  and the result might look visually reasonable, but still would be a severe "under-fit").
So the bottom line, as this example shows, is that you need to have some assumptions about the inherent uncertainties in the data (namely, a statistical model) and what class of models are reasonable for your data. (Note that some models, such as a simple linear or quadratic function, are more easily recognized by a human brain than others, so when you think in terms of patterns that are visually clear to you, you are in a way preferring this particular class of models).
If you can make such assumptions then you can essentially fit the same data to different models and apply some model selection criteria, which will tell you if a different model (either simpler or more complicated) describes the
data better. Note that statistical goodness-of-fit tests are always relative, either explicitly or implicitly, to some alternative models. That is, they tell you whether some alternative model describe the data better, and they can differ by which classes of alternative models are effectively considered.
