How to reconcile that Bayesian model selection considers the entire parameter space of a model? Background example: Coin Toss
This is the standard example for Bayesian model selection (see, e.g., here). If you know this example, there is nothing new here:
We want to find out whether a coin is fair or loaded.
We toss the coin a certain number of times and compare:

*

*$\mathcal{M}_\text{fair}$:  a fair coin toss,

*$\mathcal{M}_\text{loaded}$: a coin toss with an unknown heads probability $θ∈[0,1]$.

Now, assuming equal priors for both models, we have:
$$
p \left( \mathcal{M}_\text{loaded} \middle| \mathcal{D} \right) 
\propto
p \left( \mathcal{D} \middle | \mathcal{M}_\text{loaded} \right)
=
\int_0^1 \left( \mathcal{D} \middle | θ, \mathcal{M}_\text{loaded} \right)
p \left( θ \middle | \mathcal{M}_\text{loaded} \right)
\mathrm{d} θ,
$$
where $\mathcal{D}$ is my coin-tossing data.
With other words, we are averaging the likelihoods of the loaded model for all possible heads ratios.
My problem
I am generally struggling with the model selection averaging over all possible parameter combinations.
As a result the outcome may strongly depend on what parameter combinations I admit for a given model (or their priors, respectively).
In particular, it is usually not expected that a model aligns with a given dataset for all parameter combinations, but only a subspace thereof.
By contrast, the most likely choice of parameter plays no special role.
In the coin-toss example, my problem manifests as follows:
First suppose my data has a heads ratio of $0.55$ and the number of samples is such that Bayesian model selection prefers $\mathcal{M}_\text{fair}$ over $\mathcal{M}_\text{loaded}$.
Suppose further that I consider a third model $\mathcal{M}_\text{mildly loaded}$, which only admits $θ ∈ [0.3,0.7]$.
Since the above integral then excludes the least favourable $θ$ values, $\mathcal{M}_\text{mildly loaded}$ may end up being preferred over $\mathcal{M}_\text{fair}$.
This is a crucially different outcome:
Not only do I get a different answer to the initial question (“Is the coin loaded?”), but if I also end up with a different predictor for further coin tosses, whether I choose a single model for prediction ($\mathcal{M}_\text{mildly loaded}$) or perform model averaging.
Another way to look at my problem is this:
If I only have one model and compare two plausible parameters explaining the data in a Bayesian approach, I only need to consider the priors and Bayes factors of those two parameter values. The priors and Bayes factors of parameters that are totally implausible (given the data) don’t affect my analysis. So far so good. Now, in Bayesian model selection, the priors and Bayes factors for all parameters suddenly matter.
I consider these properties of Bayesian model selection strongly undesirable.
Is my line of thought faulty or is this a known problem and if so, is there any justification or reconciliation?
My thoughts so far

*

*I am pretty sure I understand the math and arguments behind the equations. I am just running into contradictions and undesired behaviour.


*One possible reconciliation I considered is that since $\mathcal{M}_\text{mildly loaded} ⊂ \mathcal{M}_\text{loaded}$, the former should have a smaller prior, but then analogously I have $\mathcal{M}_\text{fair} ⊂ \mathcal{M}_\text{mildly loaded} ⊂ \mathcal{M}_\text{loaded}$ and the literature usually assumes equal model priors.


*Accounting all possible parameter sets cannot be justified by preventing overfitting. For example, $\mathcal{M}_\text{mildly loaded}$ and $\mathcal{M}_\text{loaded}$ are as prone to overfitting as long as within the overlap of $θ$.


*There is no clear way on how to choose the competing models. In the coin example, what keeps me from considering $\mathcal{M}_\text{head-loaded}$ and $\mathcal{M}_\text{tail-loaded}$ as separate models?


*I vaguely guess that the problem lies in the interpretation of $p \left( \mathcal{M}_\text{loaded} \middle| \mathcal{D} \right)$ and similar quantities, i.e., what the probability of a model means and whether it’s a desirable quantity for many applications at all.


*The Jeffreys–Lindsey paradox seems to be related and considers a similar situation. However, my issues arise without comparing to a frequentist approach.

Not a Bayesian expert but I think the term to focus on here is $p(\theta|\mathcal{M}_\text{loaded})$.
This is the probability distribution of how the coin might be loaded. We might suggest that this distribution is flat (all weightings equally likely) except for $\theta=0.5$ (a fair coin).
In your first example, the data might closely resemble a fair coin so $\mathcal{M}_\text{fair}$ could be preferable to a loaded coin since there is significant probability that, if the coin is loaded, it will have a value for $\theta$, which makes the observations unlikely.
In the second example, you are modifying the probability distribution by removing values outside $[0.3, 0.7]$ and so the integral is focused towards values, where the observed data is more likely.
One more point: You say that $\mathcal{M}_\text{fair}\subset \mathcal{M}_\text{loaded}$. But this is not correct as $\theta=0.5$ is not a loaded coin. Rather $\mathcal{M}_\text{fair} \cup \mathcal{M}_\text{loaded} $ cover the entire model space.
