Is machine learning all about hyperparameter tuning? I understand the view that ML is a big optimization problem where we are trying to minimize the loss function and achieve the most optimal solution given the input. To achieve that we are feeding a loss function (let's say accuracy) and an optimizer (let's say stochastic gradient descent) which is helpful for the model to tune the parameters if it is a parametric learner, unlike kNN. But after all, the loss function, optimizer, decision boundary shape in SVM, hidden layer count in NN, maximum depth of a tree in bagging or the base estimator in boosting are all hyperparameters that the user needs to tune considering bias and variance trade-off.
Assuming that we have unlimited resources, can't we just find the strongest model by using a large GridSearchCV with many hyperparameter combinations? It boils down to this: Is ML all about hyperparameter tuning? If not, what am I missing?
I am also asking for cases where interpretability is not that important and the sole purpose is achieving the highest testing score.
Thanks in advance!

It depends on your definition of "hyperparameters": Is the model itself a hyperparameter? The loss function? The choice of input features? The preprocessing method(s)? Etc.
If yes, then your view can be justified: Application of Machine Learning boils down to finding an optimal set of "hyperparameters". There is, however, a theory behind these algorithms, and it definitely involves more. People who invented LSTM or SVM certainly did more than just "hyperparameter tuning", at least in my view. But, you might find a definition of "hyperparameters" by which any activity which involves data processing with quantifiable results is "hyperparameter tuning".
