What is wrong with treating everything as a hyperparameter? I've seen a number of questions like this asking whether certain parameters can be treated as a hyperparameter. Why can't we just treat everything as a hyperparameter? I understand that this is an extreme stance, but the way I see it, hyperparameters (along with the parameters) ultimately determines the set of all possible models that we can fit the data against. For example, if we set a low learning rate, the set of models it can explore can be a superset of that when we set a high learning rate.

I'd think about this from a practical perspective, for example a simple supervised classification task.
For this, one would normally chose a model to start with based on some heuristic about data size, shape, and quality. Said model will be parameterized, with our aim being to learn a good set of parameters to predict the class of novel examples drawn from the same distribution as the training data.
As you say, it would be perfectly possible to learn the entire set of parameters for said model using some kind of hyperparameter optimization framework. But this would be an incredibly inefficient way of training your classifier, as you treat the entire function as a black box. The classifier you've chosen will probably come with its own optimization function that aims to produce the lowest possible error on a training set, usually using some kind of feedback mechanism to update the parameters based on the quality of the predictions it is producing.
Your choice of model was a prior you imposed, but that model probably has parameters that can be used to define it but that can't be learned by the standard training algorithm for that model. Example: the number of trees in a random forest. So we need some mechanism to chose these 'hyperparameters', which are hopefully few in number. Of course the space of hyperparameters is essentially infinite so we come to some reasonable balance based on a compute/time budget, and evaluate various settings of the parameters (probably using cross validation) to find a good model for the task at hand.
