Is it true that the type of ML model used is irrelevant? I am training a model on a dataset and all types of relevant algorithms I have used converge close to the same accuracy score, meaning that no one is significantly better performing than the other. For example, if you're training a random forest and a neural network on MNIST, you'll observe an accuracy score of around 98%. Why is this the case, that bottlenecks in performance seem to be dictated by input data rather than the choice of the algorithm?

The answer to "Is it true that the type of ML model used is irrelevant?" is most definitely "No". While other answers make a reasonable points e.g. about value of collecting new data vs. selecting between models, I feel there is one point missing. Namely, your observation that "model is irrelevant" is suffering from observation bias: you are training models on MNIST that are known to perform well on this type of data. This may give false impression that the type of model doesn't matter. There will be a large number of algorithms/models that do very poorly on MNIST, try for instance a very simple linear regression and compare that to random forest and neural nets. My intuition (no rigorous evidence) is that certain algorithms also do similarly on certain datasets; I have often observed simple neural nets (MLPs) and Random forests / xgboost to be a case in point. But of course if we had more complex image data then all these methods would do much worse than convolutional NN's, and so on.
Edit: I would advise against adopting the general notion that most problems can just be solved/made progress on by collecting more data. There is a plethora of applied ML problems where improvements in model architecture, optimization etc. can yield large benefits.
