  During the past decade, machine learning has become extremely popular and can
be found in many aspects of our every day life. Nowayadays with explosion of
data while rapid growth of computation capacity, Distributed Deep Neural
Networks (DDNNs) which can improve their performance linearly with more
computation resources, have become hot and trending. However, there has not
been an in depth study of the performance of these systems, and how well they
scale.
  In this paper we analyze CNTK, one of the most commonly used DDNNs, by first
building a performance model and then evaluating the system two settings: a
small cluster with all nodes in a single rack connected to a top of rack
switch, and in large scale using Blue Waters with arbitary placement of nodes.
Our main focus was the scalability of the system with respect to adding more
nodes. Based on our results, this system has an excessive initialization
overhead because of poor I/O utilization which dominates the whole execution
time. Because of this, the system does not scale beyond a few nodes (4 in Blue
Waters). Additionally, due to a single server-multiple worker design the server
becomes a bottleneck after 16 nodes limiting the scalability of the CNTK.

Distributed, Parallel, and Cluster Computing