  Artificial intelligence has made remarkable progress in handling complex
tasks, thanks to advances in hardware acceleration and machine learning
algorithms. However, to acquire more accurate outcomes and solve more complex
issues, algorithms should be trained with more data. Processing this huge
amount of data could be time-consuming and require a great deal of computation.
To address these issues, distributed machine learning has been proposed, which
involves distributing the data and algorithm across several machines. There has
been considerable effort put into developing distributed machine learning
algorithms, and different methods have been proposed so far. We divide these
algorithms in classification and clustering (traditional machine learning),
deep learning and deep reinforcement learning groups. Distributed deep learning
has gained more attention in recent years and most of the studies have focused
on this approach. Therefore, we mostly concentrate on this category. Based on
the investigation of the mentioned algorithms, we highlighted the limitations
that should be addressed in future research.

Machine Learning;Distributed, Parallel, and Cluster Computing