Is early stopping a reasonable method to prevent overfitting in Machine Learning algorithms? I'm new to Machine learning and have just come across early stopping criteria. I understand it uses a validation set to measure how the accuracy (or any score) improves over iteration (while training). However, what confuses me is that most ML algorithms I came across, such as SVM, Linear Regression etc are convex optimization problems, i.e, the local optimum is the global optimum. Does early stopping really work on those problems?
Also, Is early stopping criteria useful for algorithms such as Decision Trees, Random Forests, Adaboost, etc, (Tree-based methods) where we may not ideally find the best optimum?

You're right, you would not normally use early stopping for standard forms of regression or SVMs. In part, you are - to some extent -  avoiding overfitting for those models by the natural limitations of these models (they only ever get a complex as you specify them to be).
It's much more typically used for things like neural networks and gradient boosting (XGBoost, LightGBM etc.). These models will keep on making the solution more complex the more iterations you do, can approximate arbitrarily complex functions and - given enough features and time - overfit as much as you like (up to and including memorising the training data). I.e. you need to somehow stop training before you overfit and early stopping is an obvious way.
