Convolutional neural networks and recurrent neural networks are designed with network structures well suited to the nature of spacial and sequential data respectively. However, the structure of standard feed-forward neural networks (FNNs) is simply a stack of fully connected layers, regardless of the feature correlations in data. In addition, the number of layers and the number of neurons are manually tuned on validation data, which is time-consuming and may lead to suboptimal networks. In this paper, we propose an unsupervised structure learning method for learning parsimonious deep FNNs. Our method determines the number of layers, the number of neurons at each layer, and the sparse connectivity between adjacent layers automatically from data. The resulting models are called Backbone-Skippath Neural Networks (BSNNs). Experiments on 17 tasks show that, in comparison with FNNs, BSNNs can achieve better or comparable classification performance with much fewer parameters. The interpretability of BSNNs is also shown to be better than that of FNNs.
The main strengths of the paper are the supporting experimental results in comparison to plain feed-forward networks (FNNs).  The proposed method is focused on discovering sparse neural networks.  The experiments show that sparsity is achieved and still the discovered sparse networks have comparable or better performance compared to dense networks.
The main weakness of the paper is lack of cohesion in contributions and difficulty in delineating the scope of their proposed approach.
Below are some suggestions for improving the paper:
Can you enumerate the paper’s contributions and specify the scope of this work?  Where is this method most applicable and where is it not applicable?
Why is the paper focused on these specific contributions?  What problem does this particular set of contributions solve that is not solvable by the baselines?  There needs to be a cohesive story that puts the elements together.  For example, you explain how the algorithm for creating the backbone can use unsupervised data.  On the other hand, to distinguish this work from the baselines you mention that this work is the first to apply the method to supervised learning problems.
The motivation section in the beginning of the paper motivates using the backbone structure to get a sparse network.  However, it does not adequately motivate the skip-path connections or applications of the method to supervised tasks.
Is this work extending the applicability of baselines to new types of problems?  Or is this work focused on improving the performance of existing methods?  Answers to these questions can automatically determine suitable experiments to run as well.  It's not clear if Pruned FNNs are the most suitable baseline for evaluating the results.  Can your work be compared experimentally with any of the constructive methods from the related work section?  If not, why?
When contrasting this work with existing approaches, can you explain how existing work builds toward the same solution that you are focusing on?  It would be more informative to explain how the baselines contribute to the solution instead of just citing them and highlighting their differences.
Regarding the experimental results, is there any insight on why the dense networks are falling short?  For example, if it is due to overfitting, is there a correlation between performance and size of FNNs?  Do you observe a similar performance vs FNNs in existing methods?  Whether this good performance is due to your contributions or due to effectiveness of the baseline algorithm, proper analysis and discussion is required and counts as useful research contribution.