Understanding Dropout
Dropout is a relatively new algorithm for training neural networks which relies on stochastically "dropping out" neurons during training in order to avoid the co-adaptation of feature detectors. We introduce a general formalism for studying dropout on either units or connections, with arbitrary probability values, and use it to analyze the averaging and regularizing properties of dropout in both linear and non-linear networks. For deep neural networks, the averaging properties of dropout are characterized by three recursive equations, including the approximation of expectations by normalized weighted geometric means. We provide estimates and bounds for these approximations and corroborate the results with simulations. Among other results, we also show how dropout performs stochastic gradient descent on a regularized error function.
This work aims to shed light to the novel neural 
      network algorithm of dropout, but grounding it to theory and through this 
      provide guidance of how relevant parameters should be chosen to achieve a 
      good performance. Quality: The quality of the work is high, the 
      quality of the manuscript can be however, improved: there are several 
      typos, or missing symbol definitions. A summary of the findings at the end 
      would also enhance the readability of the manuscript. Clarity: The 
      paper is overall clearly written subject to typos, definitions and summary 
      missing (see above). A non-exhaustive list of suggested corrections is 
      given below: -Introduction, line 7, there may be a typo/ ambiguous 
      -Eq (1): l needs to be defined -3.1, n needs to be defined. Is 
      stochasticity is S due to dropout? Please explain m=2^n (sems to be the 
      number of different inputs in the network). Typo in equation 7 -5.1 
      typo in eq 25, typo right after eq 28 (an extra fullstop) -5.2 first 
      line: in the definition of the sigmoidal a + is missing. -the caption 
      layout of the figures is non consistent -references in the text start 
      from number [6] -A summary /conclusions section is missing 
      Originality: To the best of my knowledge, the paper is 
      original, providing answers to some key questions regarding the setup of a 
      network with dropout. Significance: In my view the article is 
      significant, and I was wondering if parallels could be drawn between the 
      dropout algorithm, applied to the connecting weights rather than the 
      nodes, with the release probability of biological synapses, as those may 
      be unreliable, effectively implementing a “dropout” 
      mechanism.