KNOWLEDGE FLOW: IMPROVE UPON YOUR TEACH- ERS
A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.
This paper proposes to feed the representations of various external "teacher" neural networks of a particular example as inputs to various layers of a student network. 
The idea is quite intriguing and performs very well empirically, and the paper is also well written.  While I view the performance experiments as extremely thorough, I believe the paper could possibly use some additional ablation-style experiments just to verify the method actually operates as one intuitively thinks it should.   
Other Comments:
- Did you verify that in Table 3, the p_w values for the teachers trained on the more-relevant C10/C100 dataset are higher than the p_w value for the teacher trained on the SVHN data?  It would be interesting to see the plots of these p_w over the course of training (similar to Fig 1c) to verify this method actually operates as one intuitively believes it should.
- Integrating the teacher-network representations into various hidden layers of the student network might also be considered some form of neural architecture search (NAS)  (by including parts of the teacher network into the student architecture). 
See for example the DARTS paper: https://arxiv.org/abs/1806.09055
which similarly employs mixtures of potential connections.  
Under this NAS perspective, the dependence loss subsequently distills the optimal architecture network back into the student network architecture.
Have you verified that this method is not just doing NAS, by for example, providing a small student network with a few teacher networks that haven't been trained at all? (i.e. should not permit any knowledge flow)
- Have the authors considered training the teacher networks jointly with the student? This could be viewed as teachers learning how to improve their knowledge flow (although might require large amounts of memory depending on the size of the teacher networks).
- Suppose we have an L-layer student network and T M-layer teacher networks.
Does this imply we have to consider O(L*M*T) additional weight matrices Q?
Can you comment on the memory requirements?
- The teacher-student setup should be made more clear in Tables 1 and 2 captions (took me some time to comprehend).
- The second and third paragraphs are redundant given the Related Work section that appears later on. I would like to see these redundancies minimized and the freed up  space used to include more results from the Appendix in the main text.