LEARNING IMPLICITLY RECURRENT CNNS THROUGH PARAMETER SHARING
We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Compared to traditional CNNs, we demonstrate substantial parameter savings on standard image classification tasks, while maintaining accuracy.
Our simple parameter sharing scheme, though defined via soft weights, in practice often yields trained networks with near strict recurrent structure; with negligible side effects, they convert into networks with actual loops. Training these networks thus implicitly involves discovery of suitable recurrent architectures. Though considering only the design aspect of recurrent links, our trained networks achieve accuracy competitive with those built using state-of-the-art neural architecture search (NAS) procedures.
Our hybridization of recurrent and convolutional networks may also represent a beneficial architectural bias. Specifically, on synthetic tasks which are algorithmic in nature, our hybrid networks both train faster and extrapolate better to test examples outside the span of the training set.
The manuscript introduces a novel and interesting approach to weight sharing among CNNs layers, by learning linear combinations of shared weight templates. This allows parameter reduction, better sample efficiency. Furthermore, the authors propose a very simple way to inspect which layers choose similar combinations of template, as well as to push the network toward using similar combinations at each layer. This regularization term has a clear potential for computation reuse on dedicated hardware. The paper is well written, the method is interesting, the results are convincing and thoroughly conducted. I recommend acceptance.
1) It would be interesting to explore how often the layer parameters converge to similar weights and how similar. To this end I suggest to plot a 2d heatmap representing the similarity matrices between every pair of layers.
2) Figure 1 is not of immediate interpretability. Especially for the middle figure, what does the dotted box represent? What is the difference between weights and templates? Also it’s unclear which of the three options corresponds to the proposed method. I would have thought the middle one, but the text seems to indicate it is the rightmost one instead.
3) How are the alphas initialized? How fast are their transitions? Do they change smoothly over training? Do they evolve rapidly and plateau to a fixed value or keep changing during training? It would be really interesting to plot their value and discuss their evolution.
4) While the number of learned parameters is indeed reduced when the templates are shared among layers - which could lead to better sample efficiency - I am not sure whether the memory footprint on GPU would change (i.e., I believe that current frameworks would allocate the same kernel n-times if the same template was shared by n layers, but I am not certain). Although the potential reduction of the number of trainable parameters is an important result by itself, I wonder if what you propose would also allow to run bigger models on the same device or not, without heavy modifications of the inner machineries of pyTorch or Tensorflow.  Can you comment on this? Also note that the soft sharing regularization scheme that you propose can be of great interest for FPGA hardware implementations, that benefit a lot from module reuse. You could mention that in the paper.
5) Sec 4.1, the number of layers in one group is defined as (L-4)/3. It’s unclear to me where the 4 comes from. Also on page 6, k = (L-2)/3 - 2 is said to set one template per layer. I thought the two formulas would be the same in that case. What am I missing? Is it possible that one of the two formulas contain a typo (I believe that at the very least it should be either (L-2) or (L-4) in both cases)?
6) Sec 4.1, I find the notation SWRN-L-w-k and SWRN-L-w confusing. My suggestion is to set k to be the total number of templates (as opposed to the number of templates *per group of layers*), which makes it much easier to relate to, and most importantly allows for an immediate comparison with the capacity of the vanilla model. As a side effect, it also makes it very easy to spot the configuration with one template per layer (SWRN-L-w-L) thus eliminating the need for an ad-hoc notation to distinguish it.
7) The authors inspect how similar the template combination weights alphas are among layers. It would also be interesting to look into what is learned in the templates. CNN layers are known to learn peculiar and somewhat interpretable template matching filters. It would be really interesting to compare the filters learned by a vanilla network and its template sharing alternative. Also, I would welcome an analysis of which templates gets chosen the most at each layer in the hierarchy. It would be compelling if some kind of pattern of reuse emerged from learning.
8) Sec 4.4, it is unclear to me what can be the contribution of the 1x1 initial convolution, since it will see no context and all the information at the pixel level can be represented by a binary bit. Also, are the 3x3 convolutions “same” convolutions? If not, how are the last feature maps upscaled to be able to predict at the original resolution?
9) At the end of sec 4.4 the authors claim that the SCNN is “also advantaged over a more RNN-like model”. I fail to understand how to process this sentence, but I have a feeling that it’s incorrect to make any claims to the performance of “RNN-like models” as such a model was not used as a baseline in the experiments in any way. Similarly, in the conclusions I find it a bit stretched to claim that you can “gain a more flexible form of behavior typically attributed to RNNs”. While it’s true that the proposed network can in theory learn to reuse the same combination of templates, which can be mapped to a network with recursion, the results in this direction don’t seem strong enough to draw any conclusion and a more in-depth comparison against RNN performance would be in order before making any claim in this direction.
MINOR
- Sec3: I wouldn’t say the parameters are shared among layers in LSTMs, but rather among time unrolls.
- One drawback of the proposed method is that the layers are constrained to have the same shape. This is not a major disadvantage, but is still a constraint that would be good to make more explicit in the description of the model.
- Sec3, end of page 3: does the network reach the same accuracy as the vanilla model when k=L? Also, does the network use all the templates? How is the distribution of the alpha weights across layers in this case?
- Sec3.1, the V notation makes the narrative unnecessarily heavy. I suggest to drop it and refer directly to the templates T. Also the second part of the section, with examples of templates, doesn’t add much in my opinion and would be better depicted with a figure.
- Sec3.1, the e^(i) notation can be confused with an exp. I suggest to replace it with the much more common 1_{i=j}.
- Figure 2 depicts the relation between the LSM matrix and the topology of the network. This should be declared more clearly in the caption, in place of the ambiguous “capturing implicit recurrencies”. Also, the caption should explain what black/white stand for as well, and possibly quickly describe what the LSM matrix is. Also, it would be more clear that the network in the middle is equivalent to that on the right if the two were somehow connected in the figure. To this end they could, e.g., share a single LSM matrix among them. Finally, if possible try and put the LSM matrices on top of the related network, so that it’s clear which network they refer to. Sec 3.2 should also refer to Fig2 I believe.
- Table 1: I suggest to leave the comment on the results out of the caption, since it’s already in the main text.
- Table 2: rather than using blue, I suggest to underline the overall best results, so that it’s visible even if the paper is printed in B&W.
- Fig 3, I would specify that it’s better viewed in color
- Discussion: I feel the discussion of Table 1 is a bit difficult to follow. It could be made easier by reporting the difference in test error against the corresponding vanilla model (e.g., “improves the error rate on CIFAR10 by 0.26%”, rather than reporting the performance of both models)
- Fig 4, are all the stages the same and is the network in the left one such stages? If so, update the caption to make it clear please.
- Fig 4, which lambda has been used? Is it the same for all stages?
- Fig 5, specify that the one on the right is the target grid. Also, I believe that merging the two figures would make it easier to understand (e.g., some of the structure in the target comes from how the obstacles are placed, which requires to move back and forth from input to target several times to understand)
- Sec 4.4, space permitting, I would like to see at least one sample of what kind of shortest path prediction the network can come up with.
A few typos:
    * End of 3.2: the closer elements -> the closer the elements
    * Parameter efficiency: the period before re-parametrizing should probably be a comma?
    * Fig 4, illustration of stages -> illustration of the stages
    * End of pag7, an syntetic -> a syntetic