 The success of deep learning would not have been possible without specialized Graphics Processing Units (GPUs), massively-parallel processors optimized for the computational burden of training large nets. However, the crucial innovations behind deep learning’s successes lie in network architecture. Although the literature describes a bewildering set of variations in deep net design (Schmidhuber 2015), there are some common themes that help define the paradigm.