What is wrong with treating everything as a hyperparameter? I've seen a number of questions like this asking whether certain parameters can be treated as a hyperparameter. Why can't we just treat everything as a hyperparameter? I understand that this is an extreme stance, but the way I see it, hyperparameters (along with the parameters) ultimately determines the set of all possible models that we can fit the data against. For example, if we set a low learning rate, the set of models it can explore can be a superset of that when we set a high learning rate.

Treating "everything" as a hyperparameter leads to an infinite regress of priors
In principle, you can take any constant in a distribution that has an allowable range, and you can then treat it like a conditioning random variable.  Consequently, in principle you can always have more hyperparameters in your analysis if you want to.  But you have to stop somewhere.
Treating a formerly fixed quantity in a prior distribution as a hyperparameter means that you are changing your prior distribution.  To see this, suppose you have a prior for $\theta$ using some constant $\phi$.  If you treat $\phi$ as a hyperparameter with density $f$ then you get the following change in your (marginal) prior for your parameter:
$$\begin{matrix}
& & & \text{Prior} \\[6pt]
\text{Known constant } \phi & & & \pi(\theta|\phi) \\[6pt]
\text{Hyperparameter } \phi & & & \pi(\theta) = \int \pi(\theta|\phi) f(\phi) d \phi \\[6pt]
\end{matrix}$$
Every time we take a fixed quantity in the prior and treat it as a hyperparameter, we change the (marginal) prior.  Usually this change makes the prior become more diffuse, because of the additional uncertainty in relation to a quantity it depends on.  If we were to try to "treat everything as a hyperparameter" that would just mean that we would construct an infinite regress of prior distributions, as we take more and more quantities and assign them a hyperprior, thereby changing the (marginal) prior.  You would never get to a point where you have exhausted all quantities that could be generalised to hyperparameters, so you would never get to an endpoint giving you a prior distribution to use in your analysis.
