Why do we fix parameters and metaparameters differently? To optimize parameters of a model we minimize a loss function.
To optimize metaparameters we look at a loss/metric on a validation set ( or, if we are worried to overfit on the validation set, we do cross validation ).
(Than we check performance on the test set.)
Why this difference ? Couldn't we consider metaparameters and parameters on the same ground and optimize both of them on the training set ? Is there some fundamental mistake with this approach? What would be the consequence of such a choice ?
My only explanation is that we can get an estimate of the generalization error and it is best to use this to select an optimal model with smaller overfitting. One could do the same with parameters but would simply be too expensive (e.g. it is too expensive to consider in a neural net half of the weights as parameters and the other half as hyperparameters).
Opinions?

The previous post already addressed the question, so I just want to add some further points where they differ:
The parameter is dependent on the hyper-parameter. For example, in the case of support vector machines, you set hyper-parameter on the degree of polynomial kernel, only after that, you can obtain the optimal parameter, the loss function is convex and there is a unique solution for the parameter.
But is there an optimal hyper-parameter? No, here the problem is no longer convex and thus different methods (e.g. grid search, simplex) are available to search for the better hyper-parameters.
For one problem (finding parameter), there is often a unique solution, while for the other problem (finding hyper-parameter), there isn't. So they shouldn't be considered on the same ground or being optimized together.
Regarding "parameter optimization leads to overfitting, but we do not bother too much". Many methods incorporate some forms of regularization in the model, e.g. shrinking the coefficients through penalty term. But the use of validation set is also detecting an overfitting issue where the coefficients might be unnecessarily large.
