In cloud computing and high performance computing, a large job is typically divided into many small tasks for parallel execution in a distributed environment. Due to different reasons, some tasks (so-called ‘stragglers’) are considerably slower than the others, delaying the completion of the job. We propose a new machine learning approach to automatically identify and diagnose the stragglers. To first identify stragglers, an unsupervised clustering method is employed to group the tasks based on their execution time. We then use a supervised rule learning algorithm to learn diagnosis rules inferring the stragglers with their resource assignment and performance counter data. Preliminary experiments from the trace of a Google's Borg cluster demonstrate that our method is able to generate simple and easy-to-read rules with decent performance in predicting stragglers.
['Cong Li', 'Huanxing Shen', 'Tai Huang']
Learning to diagnose stragglers in distributed computing