Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of a neural network can be represented as a directed acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Despite the selection of micro node operations, macro connections among the whole network, noted as topology, largely affect the optimization process. We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. Motivated by this, we propose an novel method to optimize the topology of a neural network. The optimization space is defined as a complete graph, and through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. To extend the optimization to larger search spaces, a new series of networks, called TopoNet, are designed. To further focus on critical edges and promote generalization ability in dense topologies, an auxiliary sparsity constraint is adopted to constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with TopoNets further verify both availability and transferability of the proposed method in different tasks e.g. image classification, object detection, and face recognition.
I. Summary of the paper
This paper describes a principled strategy for searching for the most
suitable neural network architecture out of a particular class of
architectures. Specifically, the problem is framed as an optimisation
problem over a set of directed acyclic graphs (DAGs) that correspond to
potential network architectures. By optimising the edge weights of this
representation, a suitable architecture can be generated.
In addition to the aforementioned optimisation scheme, the paper also
presents a regularisation that results in *sparse* networks, i.e.
networks with a smaller number of edges. Multiple experiments on
'tuning' existing architectures on several data sets conclude the paper.
II. Summary of the review
This paper discusses a highly relevant subject, namely how to select
neural network architectures in a principled manner. While the presented
work already goes into a good direction, I cannot give it my endorsement
for acceptance because of the following reasons:
  - The paper is lacking clarity: concepts could be explained somewhat
    better, and the paper is suffering from language/grammar issues that
    make it harder to understand the contents.
  - Lack of experimental or theoretical depth: the proposed method is
    presented as-is; no theoretical analysis of its behaviour is
    performed; while this is not necessarily a problem, as there are
    several empirical experiments, the experimental section is not
    sufficiently detailed: for example, no limitations of the method are
    being discussed and the presented results are not state-of-the-art
    accuracies.
Nevertheless, I want to point out that this paper has the potential to
become an important contribution to the community; it is absolutely
clear that more principled approaches are required to select network
architectures.
In the following, I will comment on the individual aspects in more
detail.
III. Detailed comments (clarity)
- The abstract could be improved in terms of its logical flow. Instead
  of trying to introduce new terminology (macro/micro etc.) here, the
  abstract should rather state directly that this paper frames network
  architecture selection as an optimisation problem over DAGs.
- The use of topology is slightly non-standard here. What is the meaning
  behind the 'macro' and 'micro' operations? This should be explained
  somewhat better.
- Figure 1 should be extended to show an example of how the depicted
  graphs are described through the terminology mentioned in the paper.
  For example, individual edges or nodes could be highlighted and
  referred to in the text to make the 'mapping' clearer.
- I do not understand how operations such as *addition* are represented
  in the DAG. Ideally, this should also be elucidated by a figure.
- When discussing 'intervals' of residual connections, I am assuming
  that the paper refers to how many layers are skipped? If so, this
  should be mentioned and defined explicitly.
- The term 'searching space' should be replaced by 'search space', as
  the latter is more standard usage.
- I do not understand why the optimisation of the topology can decrease
  the computational burden, as claimed on p. 2. The optimisation process
  still has to be performed, just like the training of the network,
  correct? Am I misunderstanding this?
- The caption of Figure 2 could be extended; does a single node type
  mean that the complete network only consists of nodes of that type?
  Moreover, accuracies/errors should be shown in addition to the loss
  curves.
- The term 'dense connection' is vague; I think the paper should use
  'densely-connected graph' here.
- The sentence 'Among these nodes, [...]' refers to the *whole* network,
  and not to the way the output tensor $\mathbf{x}_i$ is processed. Am I
  understanding this correctly?
- I do not understand the initial sentences in Section 2.2; what is the
  meaning of 'cell' in this case?
- The term 'topological structures' should be renamed to '(sub)graphs'
  in order to improve clarity.
- I do not understand the comment on sparsity in Section 2.4. How are
  'moderate sparsity' and Figure 2 connected?
- In the algorithm, I would use 'Sparsity Type' instead of 'Sparse Type'
  to refer to the parameter.
- What does 'Complete' (without $\mathbf{\alpha}$) mean in Table 3?
- The footnote below Table 3 is not referenced anywhere in the text or
  in the table.
- In Figure 3, are the adjacency matrices consistent? What happens if
  the training process is repeated? It would be highly interesting to
  show 'averaged' matrices over multiple runs.
IV. Detailed comments (experiments & theory)
- A theoretical analysis of the proposed method would be interesting.
  Does the optimisation always converge? Are minima unique? What is the
  computational complexity?
  At least some of these aspects should be discussed.
- The limitations of the proposed method are not explained. For example,
  what is the meaning of the sentence on p. 2 about 'excluding the
  influence of the mixture of different layers/nodes'?
  It is my understanding that the proposed method can only change the
  *connections* between blocks of a network, but not the type of layers.
  Is this correct? If so, it would be a major limitation that should be
  mentioned explicitly.
- Another limitation that is not discussed is the scaling to very deep
  networks. How problematic is it to model all potential connections in
  such a network? Are there limits to the current optimisation scheme?
  This needs to be assessed in the experimental section.
- For all experimental tables, standard deviations and means should be
  provided. This is necessary in order to assess the stability of the
  proposed method, because there are multiple sources of stochasticity:
  one arising from the optimisation procedure, the other one arising
  from the training of the network itself.
- The results reported for the experiments are somewhat behind the
  state-of-the-art in terms of accuracy values. This should be stated
  more clearly; I assume that it is caused by limitations of the
  proposed method, which prohibit an application to very recent
  architectures. Is this correct? If so, it should at least be
  mentioned.
- The claim that nodes at the start of a topological ordering contribute
  more to specific stages needs to be (empirically) proven.
V. Style issues
The paper is not easy to read because of several non-standard phrases or
expressions.
- The phrase 'in topology' is often added to a sentence where it does
  not entirely make sense. For example, '[The] architecture can be
  expressed as a directed acyclic graph (DAG) in topology'. I do not
  see the necessity of adding 'in topology' here. There are other places
  at well from which I would remove this phrase.
- 'effective networks' --> 'effective network architectures'
- 'largely affects' --> 'largely affect'
- 'Motivated by which' --> 'Motivated by this'
- 'innovative method' --> 'method' (or 'novel method')
- 'as a complete graph, through' --> 'as a complete graph, and through'
- 'auxiliary sparsity constraint' --> 'an auxiliary sparsity constraint'
- 'named as TopoNet' --> 'called TopoNet'
- 'At initial periods' --> 'Previously' (I am not sure I understand
  this correctly)
- 'red signs' --> 'red arrows'
- 'for its topology' --> 'in terms of its topology' (?)
- 'both combining' --> 'both a combination' (?)
- 'number of interval' --> 'number of intervals'
- 'straight connected' --> 'directly connected'
- 'conduct transformation' --> 'performs a transformation'
- What is the meaning of the phrase 'These may cover the influence
  [...]'? Is this a reference to limitations of existing networks?
- 'opted from' --> 'chosen from'
- 'Following two simple design rules' --> 'We follow two simple design
  rules'
- '1000-dimension' --> '$1000$-dimensional'
- 'We raise two ways' --> 'We describe two ways'
- 'consited' --> 'consisted' / 'consists'
- 'deepen the depth' --> 'increase the depth'
- 'origin' --> 'original'
- 'promotions' --> 'improvements'
- 'can make more profit' --> 'can be useful to improve performance' (I
  am guessing this from the context)
- 'sparseness on representation' --> 'sparsity'
- 'Adaptive one' --> 'The adaptive one'
- 'At the fore' --> 'At the beginning/start'
- I do not understand the sentence about the 'free lunch'. Does it refer
  to the fact that some connections can still be removed from the
  network without decreasing accuracy?
- 'less computation costs' --> 'lower computation costs'
- 'shortcut offers' --> 'shortcuts offer'
- 'benefits optimization' --> 'benefit optimization'
- 'feasible way to the optimization' --> 'feasible way for the optimization'
- Some references in the bibliography are not capitalised consistently