Compressed forms of deep neural networks are essential in deploying large-scale computational models on resource-constrained devices. Contrary to analogous domains where large-scale systems are build as a hierarchical repetition of smallscale units, the current practice in Machine Learning largely relies on models with non-repetitive components. In the spirit of molecular composition with repeating atoms, we advance the state-of-the-art in model compression by proposing Atomic Compression Networks (ACNs), a novel architecture that is constructed by recursive repetition of a small set of neurons. In other words, the same neurons with the same weights are stochastically re-positioned in subsequent layers of the network. Empirical evidence suggests that ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks (88× to 1116× reduction) with only a fractional deterioration of classification accuracy (0.15% to 5.33%). Moreover our method can yield sub-linear model complexities and permits learning deep ACNs with less parameters than a logistic regression with no decline in classification accuracy.
This paper proposes a new way to create compact neural net, named Atomic Compression Networks (ACN). An immediate related work is LayerNet, where a deep neural net is created by replicating the same layer. Here, this paper extends replication down to the neuron level. 
I am leaning towards rejecting this paper because the experimental setup is not well justified and a few important details are missing before conclusions can be drawn. I would like to ask a few clarification questions. Depending on the authors’ answers, I might be willing to adjust my rating. 
(1) Is there missing a delta in the first half of line 6 in Algorithm 1? 
(2) Throughout the experiments, for the same hyperparameter (e.g. Table 4 in A.2) do you run Algorithm 1 more than once and select the best sample architecture? If the answer is yes, summarizing all masks as one parameter will not be reasonable. Given a yes answer, I would also like to ask if the same number of samples have been considered for FC (for the same hyperparameter). 
(3) Is there any intuition behind why FC does a much worse job of fitting curves than ACN with much less parameters? This refers to Fig. 2, if we compare FC with 41 parameters to ACN with 18 parameters. I am confused because MSE on sampled points often goes down when we increase the number of parameters for the application of curve fitting.
(4) Convolution can be thought of as a special case of ACN. ConvNet is the default architecture for working on image datasets. Since MNIST and CIFAR are considered, why not also compare to ConvNet?
(5) The claims that “ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks with only a fractional deterioration of classification accuracy” is quite misleading. Given fully-connected neural networks achieve up to 528 times with also a fractional deterioration (Sec. 4.3), by presumably having a shallower architecture.