How to use priors on the parameter number with an information criterion (AIC, BIC, …)? Example
The example is made up because I hope that it’s more accessible than my actual problem.
I want to determine the number of planets of a star.
I have:

*

*data for some astronomical observable of that star, e.g., an intensity time series,

*a model that describes the observable with the sizes of the star’s planets as parameters,

*goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,

*prior data on how likely a given number of planets is,

*no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.
Question
If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting?
In the example, how do I penalise an extra planet?
I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.
Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood.
My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.
On the other hand, I was thinking: What if the ratio between the prior probabilities of one and no planet was higher than the parameter penalty imposed by the information criterion?
For example, in case of the AIC, what if one planet were more probable than no planet by a factor $e$ (from prior insights)?
In that case, my approach would always favour one planet over none – unless I implement a penalty, lower prior, or similar for very small planets.

You are mixing two different concepts: model selection and regularization.

*

*AIC and BIC are used for model selection. Model selection is not that different between frequentist and Bayesian settings, though they may use different tools and methods. You can easily search for Bayesian model selection to learn more.

*Another thing is regularization. In a frequentist setting, you could use regularization to penalize the number of parameters (e.g. ridge or lasso regression), the same is done in a Bayesian setup with choosing sparsity introducing priors that drag the parameters towards zero. You either need to choose the priors based on what you know about the problem or choose a hyperprior that would be used to learn the priors.

