Testing the similarity of datasets containing many variables to asses model predictive performance I have 2 large datasets containing many variables, A & B, where A is taken from period t and B is taken from period t + n. I have a model that was trained on dataset A whose predictive performance has now deteriorated on dataset B.
I want to test the similarity of the 2 datasets in order to assess whether the decrease in performance of the model is due to;

*

*A statistically significant change in the underlying
data, i.e., there is a fundamental change in the data-
generating process and dataset A no longer contains
predictive information for the behaviour


*The deterioration is due to overfitting
Firstly, are points 1 & 2 essentially the same thing?
If not, what are some methods to investigate this? I would be interested in hearing if there is a bayesian approach as well.

I think these cases are both the same, if the data set has changed, then due to over-fitting on the previous situation, the model performance deteriorates.
I guess you could ask yourself if the deterioration is statistically significant. This is a statistical test depending on how many samples you've calculated your accuracy for and how large the deviation is.
