How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency? There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind:

*

*When optimizing a formulation such as a chemical formula or a mixture of chemicals for a target application, the target property doesn't change if you switch the order in which the elements appear (i.e., $SiO_2$ is equivalent to $O_2Si$, 80% $H_2O$+20% $HCl$ is the same as 20% $HCl$+80% $H_2O$)$^1$.

*When optimizing a formulation, there is often a compositional (linear equality) constraint $^2$ such that the fractional contributions sum to $1$ (e.g. $x_1+x_2+x_3=1$). A degenerate dimension can be removed (retaining only the $n-1$ degrees of freedom) by representing it as a linear inequality constraint (e.g. $x_1+x_2\le1$ where $x_3=1-x_1-x_2$).

*Similarly, there are "unitless" physics-based simulations where scaling certain inputs related to size (e.g. multiplying everything by 10) results in the same output. e.g. $[x_1, x_2, x_3]$ can be reparameterized to $[r_1=\frac{x_1}{x_3}, r_2=\frac{x_2}{x_3}]$ where $x_3$ is some constant (e.g. $1$).

1. Except perhaps in the case where you're trying to model "steps" (e.g. first add 80% $H_2O$ then add 20% $HCl$) 
2. There are also cases where magnitude affects the objective rather than being solely dependent on fractional contribution. 
In the first case, an ordering constraint can be applied (i.e., impose constraints that elements have to appear in a certain order). When dataset scaling isn't an issue, a better solution might be to perform data augmentation where you give it all symmetric representations. In the second case, you can reparameterize a linear equality constraint as a linear inequality constraint. In the third case, you can reparameterize by setting one variable as a constant and dividing the other variables by the constant.
Intuitively and in general, I would expect each of these to have a non-negligible benefit to the search space efficiency. In the last two cases, there is less "hay" relative to the "number of needles" (EDIT: which become "beads" in the lower dimension). For the first case using data augmentation, when you search in one spot, you get to search in a few other specific places at "no-cost" for a fixed amount of hay.
I tried to pose the title question in a way that targets the intended audience within the character limit. If I were to ask it again:
What examples (analytical, physical, computational, etc.) are you aware of that either corroborate or contradict the assumption that addressing search space symmetry will have a non-negligible benefit to Bayesian optimization search efficiency?
Related

*

*How to handle real-world (soft) constraints in an optimization problem?

*How to implement Bayesian Optimization when there is a relationship between search space parameters?

*Methods section of https://doi.org/10.48550/arXiv.2203.12597
UPDATE: I am actively working on a manuscript related to this, and it's still really important for me to know what prior work has been done. Planning to update this with an arXiv link to the manuscript when available.

I explored this topic in the following manuscript:

Baird, S.; Hall, J. R.; Sparks, T. D. Effect of Reducible and Irreducible Search Space Representations on Adaptive Design Efficiency: A Case Study on Maximizing Packing Fraction for Solid Rocket Fuel Propellant Simulations; preprint; Chemistry, 2022. https://doi.org/10.26434/chemrxiv-2022-nz2w8.

Interestingly, we found that the most compact search space is not necessarily the most efficient, which is related to the interactions of the size-invariance degeneracy (case 3. in OP question) with other constraints.
The most efficient search space had the compositional (case 2. above) and order constraint (case 1.).
On average, the least efficient search space wasn't the least compact. It was one that had both the size reparameterization and the order reparameterization. It was surprising that when treated individually, the size and order reparameterizations did better than the combination of size + order.
The results are summarized in Figure 2:

