Published as a conference paper at ICLR 2018 BOOSTING DILATED CONVOLUTIONAL NETWORKS WITH MIXED TENSOR DECOMPOSITIONS
The driving force behind deep networks is their ability to compactly represent rich classes of functions. The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to replicate functions of another. To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones. In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways. We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks. By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency. In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not. Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy. This leads us to believe that expressive efficiency may serve a key role in developing new tools for deep network design.
(Emergency review—I have no special knowledge of the subfield, and I was told a cursory review was OK, but the paper was fascinating and I ended up reading fairly carefully)
This paper does many things. It adds to a series of publications that analyze deep network architectures as parameterized decompositions of intractably large tensors (themselves the result of discretizing the entire input-output space of the network), this time focusing on the WaveNet architecture for autoregressive sequence modeling. It shows (first theoretically, then empirically) that the WaveNet's structural assumption of a single (perfect) binary tree is holding it back, and that WaveNet-like architectures with more complex mixed tree structures perform better.
Throughout the subject is treated with a high level of mathematical rigor, while relegating proofs and detailed walkthrough explanations to lengthy appendices which I did not have time to review.
Some things I noticed:
- The notation used is mostly consistent, except for some variation between dots (e.g., in Eq. 2) and bra-kets (in Fig. 1) for inner product. While I think I'm in the minority here, I'd personally be comfortable with going a little bit further with index notation and avoiding the cohabitation of tensor and vector notation styles by using indices even for dot products; that said, either kind of vector notation (dots or brakets) is certainly acceptable too.
- There are a couple more nomenclature things that might trip up those of us in the deep learning crowd—we're used to referring to "axes" or "dimensions" of a tensor, but the tensor-analysis world apparently says "modes" (and this is called out once in a parenthetical). As "dimension" means something different to tensor folks (what DLers usually call the "size" of an axis), perhaps standardizing on the shared term "axes" would be worthwhile? Not sure if there's a distinction in the tensor world between the words "axis" and "mode."
- The baseline WaveNet is only somewhat well described as "convolutional;" the underlying network unit is not always a "size-2 convolution" (except for certain values of g) and the "size-1 convolutions" that make it up are simply linear transformations. While the WaveNet derives from convolutional sequence architectures (and the choices of g explored in the original paper derive from the CNN literature) it has at least as much in common with recursive/tree-structured network architectures like TreeLSTMs and RNTNs. In fact, the WaveNet is a special case of a recursive neural network with a particular composition function *and a fixed (perfect) binary tree structure.* As this last condition is relaxed in the present paper, making the space of networks under analysis more similar to the traditional space of recursive NNs, it might be worth mentioning this "alternative history" of the WaveNet.
- The choice of mixture nodes in Fig. 3 is a little unfortunate, because it includes all possible mixture nodes and doesn't make it as clear as the text does that a subset of these nodes can be chosen in the general case.
- While I couldn't follow some of Section 5, I'm a little confused that Theorem 1 appears at first glance to apply only to a non-generalized decomposition (a specific choice of g).
- Caffe would not have been my first choice for such a complex, hierarchically structure architecture; I imagine it forced the authors to write a significant amount of custom code.