TRAINING GROUP ORTHOGONAL NEURAL NETWORKS WITH PRIVILEGED INFORMATION
Learning rich and diverse feature representation are always desired for deep convolutional neural networks (CNNs). Besides, when auxiliary annotations are available for specific data, simply ignoring them would be a great waste. In this paper, we incorporate these auxiliary annotations as privileged information and propose a novel CNN model that is able to maximize inherent diversity of a CNN model such that the model can learn better feature representation with a stronger generalization ability. More specifically, we propose a group orthogonal convolutional neural network (GoCNN) to learn features from foreground and background in an orthogonal way by exploiting privileged information for optimization, which automatically emphasizes feature diversity within a single model. Experiments on two benchmark datasets, ImageNet and PASCAL VOC, well demonstrate the effectiveness and high generalization ability of our proposed GoCNN models.
Learning rich and diverse feature representation are always desired for deep convolutional neural networks (CNNs). Besides, when auxiliary annotations are available for specific data, simply ignoring them would be a great waste. In this paper, we incorporate these auxiliary annotations as privileged information and propose a novel CNN model that is able to maximize inherent diversity of a CNN model such that the model can learn better feature representation with a stronger generalization ability. More specifically, we propose a group orthogonal convolutional neural network (GoCNN) to learn features from foreground and background in an orthogonal way by exploiting privileged information for optimization, which automatically emphasizes feature diversity within a single model. Experiments on two benchmark datasets, ImageNet and PASCAL VOC, well demonstrate the effectiveness and high generalization ability of our proposed GoCNN models.
1 INTRODUCTION
Deep convolutional neural networks (CNNs) have brought a series of breakthroughs in image classification tasks (He et al., 2015; Girshick, 2015; Zheng et al., 2015). Many recent works (Simonyan & Zisserman, 2014; He et al., 2015; Krizhevsky et al., 2012) have observed that CNNs with different architectures or even different weight initializations may learn slightly different feature representations. Combining these heterogeneous models can provide richer and more diverse feature representation which can further boost the final performance. Such observation motivate us to directly pursue feature diversity within a single model in the work.
Besides, many existing datasets (Everingham et al., 2010; Deng et al., 2009; Xiao et al., 2010) provide more than one types of annotations. For example, the PASCAL VOC (Everingham et al., 2010) provides image level tags, object bounding box, and image segmentation masks; the ImageNet dataset (Deng et al., 2009) provide image level tags and a small portion of bounding box. Only using the image level tags for training image classification model would be a great waste on the other annotation resources. Therefore, in this work, we investigate whether these auxiliary annotations could also help a CNN model learn richer and more diverse feature representation.
In particular, we take advantage of these extra annotated information during training a CNN model for obtaining a single CNN model with sufficient inherent diversity, with the expectation that the model is able to learn more diverse feature representations and offers stronger generalization ability for image classification than vanilla CNNs. We therefore propose a group orthogonal convolutional neural network (GoCNN) model that is able to exploit these extra annotated information as privileged information. The idea is to learn different groups of convolutional functions which are “orthogonal” to the ones in other groups. Here by “orthogonal”, we mean there is no significant correlation among the produced features. By “privileged information”, we mean these auxiliary information only been used during the training phase. Optimizing orthogonality among convolutional functions reduces redundancy and increases diversity within the architecture.
Properly defining the groups of convolutional functions in the GoCNN is not an easy task. In this work, we propose to exploit available privileged information for identifying the proper groups. Specifically, in the context of image classification, object segmentation annotations which are (partially) available in several public datasets give richer information.
In addition, the background contents are usually independent on foreground objects within an image. Thus, splitting convolutional functions into different groups and enforcing them to learn features from the foreground and background separately can help construct orthogonal groups with small correlations. Motivated by this, we introduce the GoCNN architecture which explores to learn discriminative features from foreground and background separately where the foreground-background segregation is offered by the privileged segmentation annotation for training GoCNN. In this way, inherent diversity of the GoCNN can be explicitly enhanced. Moreover, benefiting from pursuing the group orthogonality, the learned convolutional functions within GoCNN are demonstrated to be foreground and background diagnostic even for extracting features from new images in the testing phase.
To the best of our knowledge, this work is the first to explore a principled way to train a deep neural network with desired inherent diversity and the first to investigate how to use the segmentation privileged information to assist image classification within a deep learning architecture. Experiments on ImageNet and PASCAL VOC clearly demonstrate GoCNN improves upon vanilla CNN models significantly, in terms of classification accuracy.
As a by-product of implementing GoCNN, we also provide positive answers to the following two prominent questions about image classification: (1) Does background information indeed help object recognition in deep learning? (2) Can a more precise annotation with richer information, e.g., segmentation annotation, assist the image classification training process non-trivially?
2 RELATED WORK
Learning rich and diverse feature representations is always desired while training CNNs for gaining stronger generalization ability. However, most existing works mainly focus on introducing handcrafted cost functions to implicitly pursue diversity (Tang, 2013), or modifying activation functions to increase model non-linearity (Jin et al., 2015) or constructing a more complex CNN architecture (Simonyan & Zisserman, 2014; He et al., 2015; Krizhevsky et al., 2012). Methods that explicitly encourage inherent diversity of CNN models are still rare so far.
Knowledge distillation (Hinton et al., 2015) can be seen as an effective way to learn more discriminative and diverse feature representations. The distillation process compresses knowledge and thus encourages a weak model to learn more diverse and discriminative features. However, knowledge distillation works in two stages which are isolated from each other and has to rely on pre-training a complicated teacher network model. This may introduce undesired computation overhead. In contrast, our proposed approach can learn a diverse network in a single stage without requiring an extra network model. Similar works, e.g. the Diversity Networks (Sra & Hosseini), also squeeze the knowledge by preserving the most diverse features to avoid harming the performance.
More recently, Cogswell et al. (2016) proposed the DeCov approach to reduce over-fitting risk of a deep neural network model by reducing feature covariance. DeCov also agrees with increasing generalization ability of a model by pursuing feature diversity. This is consistent with our motivation. However, DeCov penalizes the covariance in an unsupervised fashion and cannot utilize extra available annotations, leading to insignificant performance improvement over vanilla models (Cogswell et al., 2016).
Using privileged information to learn better features during the training process is similar in spirit with our method. Both our proposed method and Lapin et al. (2014) introduce privileged information to assist the training process. However, almost all existing works (Lapin et al., 2014; Lopez-Paz et al., 2016; Sharmanska et al., 2014) are based on SVM+ which only focuses on training a better classifier and is not able to do the end-to-end training for better features.
Several works (Andrew et al., 2013; Srivastava & Salakhutdinov, 2012) about canonical correlation analysis (CCA) for CNNs provide a way to constrain feature diversity. However, the goal of CCA
is to find linear projections for two random vectors that are maximally correlated, which is different from ours.
It is also worth to notice that simply adding a segmentation loss to image classification neural network is not equivalent to a GoCNN model. This is because image segmentation requires each pixel within the target area to be activated and the others stay silent for dense prediction, while GoCNN does not require the each pixel within the target area to be activated. GoCNN is specifically designed for classification tasks, not for segmentation ones. Moreover, our proposed GoCNN supports learning from partial privileged information wile the CNN above needs a fully annotated training set.
3 MODEL DIVERSITY OF CONVOLUTIONAL NEURAL NETWORKS
Throughout the paper, we use f(·) to denote a convolutional function (or filter) and k to index the layers in a multi-layer network. We use c(k) to denote the total number of convolutional functions at the k-th layer and use i and j to index different functions, i.e., f (k)i (·) denotes the i-th convolutional function at the k-th layer of the network. The function f maps an input feature map to another new feature map. The height and the width of a feature map output at the layer k are denoted as h(k) and w(k) respectively. We consider a network model consisting of N layers in total.
Under a standard CNN architecture, the elements within the same feature map are produced by the same convolutional function f (k)i and thus they represent the same type of features across different locations. Therefore, encouraging the feature variance or diversity within a single feature map does not make sense. In this work, our target is to enhance the diversity among different convolutional functions. Here we first give a formal description of model diversity for an N -layer CNN.
Definition 1 (Model Diversity). Let f (k)i denote the i-th convolutional function at the k-th layer of a neural network model, and then the model diversity of the k-th layer is defined as
ζ(k) , 1− 1 c(k) 2 c(k)∑ i,j=1 cor ( f (k) i , f (k) j ) . (1)
Here the operator cor(·, ·) denotes the statistical correlation.
In other words, the inherent diversity of a network model that we are going to maximize is evaluated across all the convolutional functions within the same layer.
The most straightforward way to maximize the above diversity for each layer is to directly maximize the quantity of ζ(k) during training the network. However, it is quite involved to optimize the hard diversity in (1) due to the large combination number of different convolutional functions. Thus, we propose to solve this problem by learning the convolutional functions in different groups separately. Different functions from different groups are uncorrelated to each other and we do not need to consider their correlation. Suppose the convolutional functions at each layer are partitioned into m different groups, denoted as G = {G1, . . . , Gm}. Then, we instead maximize the following Group-wise Model Diversity.
Definition 2 (Group-wise Model Diversity). Given a pre-defined group partition set G = {G1, . . . , Gm} of convolutional functions at a specific layer, the group-wise model diversity of this layer is defined as
ζ(k)g , 1− 1
c(k) 2 |G|∑ s,t=1 ∑ i∈Gs,j∈Gt cor ( f (k) i , f (k) j ) .
Instead of directly optimizing the model diversity, we consider optimizing the group-wise model diversity by finding a set of orthogonal groups {G∗1, . . . , G∗m}, where convolutional functions within each group are uncorrelated with others within different groups. In the scenario of image representation learning, one typical example of such orthogonal groups is the foreground group and background group pair — partitioning the functions into two groups and letting them learn features from foreground and background contents respectively.
In this work, we use segmentation annotation as privileged information for finding orthogonal groups of convolutional functions G∗1, . . . , G ∗ m. In particular, we derive the foreground and background segregation from the privileged information for an image. Then we partition convolutional functions at a specific layer of a CNN model into foreground and background groups respectively, and train a GoCNN model to learn the foreground and background features separately. Details about the architecture of the GoCNN and the training procedure of GoCNN are given in the following section.
4 GROUP ORTHOGONAL CONVOLUTIONAL NEURAL NETWORKS
We introduce the group orthogonal constraint to maximize group-wise diversity among different groups of convolutional functions explicitly by constructing a group orthogonal convolutional neural network (GoCNN). Details on the architecture of GoCNN are shown in Figure 1. GoCNN is built upon a standard CNN architecture. The convolutional functions at the final convolution layer are explicitly divided into two groups: the foreground group which concentrates on learning the foreground feature and the background group which learns the background feature. The output features of these two groups are then aggregated by a fully connected layer.
In the following subsections, we give more details of the foreground and background groups construction. After that, we will describe how to combine these two components and build them into a unified network architecture — the GoGNN.
4.1 FOREGROUND AND BACKGROUND GROUPS
To learn convolutional functions that are specific for foreground content of an image, we propose the following two constraints for the foreground group of functions. The first constraint forces the functions to be learned from the foreground only and free of any contamination from the background, and the second constraint encourages the learned functions to be discriminative for image classification.
We learn features that only lie in the foreground by suppressing any contamination from the background. As aforementioned, here we use the object segmentation annotations (denoted as Mask) as the privileged information in the training phase to help identify the background features where the foreground convolutional functions should not respond to. The background contamination is extracted by an extractor adopted on each feature map within the foreground group. In particular, we define an extractor ϕ(·, ·) as follows:
ϕ(f (k) i (x),Mask) , f (k) i (x) Mask, (2)
where x denotes the raw input and denotes the element-wise multiplication. In the above operator, we use the background object mask Maskb to extract background features. Each element in Maskb is equal to one if the corresponding position lies on a background object or zero otherwise. Here, we assume the masks are already re-sized to have compatible dimensionality with the output feature map f (k)i (x) by the interpolation method so that the element-wise multiplication is valid here. The extracted background features are then suppressed by a regression loss defined as follows:
min θ ∑ i ‖ϕ(f (k)i (x; θ),Maskb)‖F . (3)
Here θ parameterizes the convolution function f (k)i . Since the target value for this regression is zero, we also call it a suppression term. It will only suppress the response output by f (k)i at the locations outside the foreground.
For the second constraint, i.e., encouraging the functions to learn discriminative features, we simply use the standard softmax classification loss to supervise the learning phase.
The role of the background group is complementary to the foreground one. It aims to learn convolutional functions that are only specific for background contents. Thus, the functions within the background group have a same suppression term as in Eqn. (3), in which Maskb is replaced with Maskf to restrict the learned features to make them only lie in the background space. The Maskf is simply computed as Maskf = 1 − Maskb. Also, a softmax linear classifier is attached during training to guarantee that these learned background functions are useful for predicting image categories.
4.2 ARCHITECTURE AND IMPLEMENTATION DETAILS OF THE GOCNN
In GoCNN, the size ratio of foreground group and background group is fixed to be 3:1 during training, since intuitively the foreground contents are much more informative than the background contents in classifying images. A single fully connected layer (or multiple layers depending on the basic CNN architecture) is used to unify the functional learning within different groups and combine features learned from different groups. It aggregates the information from different feature spaces and produces the final image category prediction. More details are given in Figure 1.
Because we are dealing with the classification problem, a main classifier with a standard classification loss function is adopted at the top layer of GoCNN. In our experiments, the standard softmax loss is used for single-label image classification and the logistic regression loss is used for multiplelabel image classification, e.g., images from the Pascal VOC dataset (Everingham et al., 2010).
During the testing stage, parts unrelated to the final main output will be removed, as shown in Figure 1 (b). Therefore, in terms of testing, neither extra parameters nor extra computational cost is introduced. The GoCNN is exactly the same as the adopted CNN in the testing phase.
In summary, for an incoming training sample, it passes through all the layers to the final convolution layer. Then the irrelevant features for each group (foreground or background) will be filtered out by privileged segmentation masks. Those filtered features will then flow into a suppressor (see Eqn. (3)). For the output features from each group, it will flow up along two paths: one leads to the group-wise classifier, and the other one leads to the main classifier. The three gradients from the suppressors, the group-wise classifiers and the main classifier will be used for updating the network parameters.
Applications with Incomplete Privileged Information Our proposed GoCNN can also be applied for semi-supervised learning. When only a small subset of images have the privileged seg-
mentation annotations in a dataset, we simply set the segmentations of images without annotations to be Maskf = Maskb = 1, where 1 is the matrix with all of its elements being 1. In other words, we disable both the suppression terms (ref. Eqn. (3)) on foreground and background parts as well as the extractors on the back propagation path. By doing so, fully annotated training samples with privileged information will supervise GoCNN to learn both discriminative and diverse features while the samples with only image tags only guide GoCNN to learn category discriminative features.
5 EXPERIMENTS
5.1 EXPERIMENT SETTINGS AND IMPLEMENTATION DETAILS
Datasets We evaluate the performance of GoCNN in image classification on two benchmark datasets, i.e., the ImageNet (Deng et al., 2009) dataset and the Pascal VOC 2012 dataset (Everingham et al., 2010).
• ImageNet ImageNet contains 1,000 fine-grained classes with about 1,300 images for each class and 1.2 million images in total, but without any image segmentation annotations. To collect privileged information, we randomly select 130 images from each class and manually annotate the object segmentation masks for them. Since our focus is on justifying the effectiveness of our proposed method, rather than pushing the state-of-the-art, we only collect privileged information for 10% data (overall 130k training images) to show performance improvement brought by our model. We call the new dataset consisting of these segmented images as ImageNet-0.1m. For evaluation, we use the original validation set of ImageNet which contains 50,000 images. Note that neither our baselines nor the proposed GoCNN needs segmentation information in testing phase.
• PASCAL VOC 2012 The PASCAL VOC 2012 dataset contains 11,530 images from 20 classes. For the classification task, there are 5,717 images for training and 5,823 images for validation. We use this dataset to further evaluate the generalization ability of different models including GoCNN trained on the ImageNet-0.1m: we pre-train the evaluated models on the ImageNet0.1m dataset and fine-tune them using the logistic regression loss on PASCAL VOC 2012 training set. We evaluate their performance on the validation set.
The Basic Architecture of GoCNN In our experiments, we use the ResNet (He et al., 2015) as the basic architecture to build GoCNN. Since the deepest ResNet contains 152 layers which will cost several weeks to train, we choose a light version of architecture (ResNet-18 (He et al., 2015)) that contains 18 layers as our basic model for most cases. We also use the ResNet-152 (He et al., 2015) for experiments on the full ImageNet dataset. The final convolution layer gives a 7× 7 output and is pooled into a 1× 1 feature map by average pooling. Then a fully connected layer is added to perform linear classification. The used loss function for the single class classification on ImageNet dataset is the standard softmax loss. When performing multi-label classification on PASCAL VOC, we use the logistic regression loss.
Training and Testing Strategy We use MXNet (Chen et al., 2015) to conduct model training and testing. The GoCNN weights are initialized as in He et al. (2015) and we train GoCNN from scratch. Images are resized with a shorter side randomly sampled within [256, 480] for scale augmentation and 224×224 crops are randomly sampled during training (He et al., 2015). We use SGD with base learning rate equal to 0.1 at the beginning and reduce the learning rate by a factor of 10 when the validation accuracy saturates. For the experiments on ResNet-18 we use single node with a minibatch size of 512. For the ResNet-152 we use 48 GPUs with mini-batch size of 32 for each GPU. Following He et al. (2015), we use a weight decay of 0.0001 and a momentum of 0.9 in the training.
We evaluate the performance of GoCNN on two different testing settings: the complete privileged information setting and the partial privileged information setting. We perform 10-crop testing (Krizhevsky et al., 2012) for the complete privileged information scenario, and do a single crop testing for the partial privileged information scenario for convenience.
Compared Baseline Models Our proposed GoCNN follows the Learning Using Privileged Information (LUPI) paradigm (Lapin et al., 2014), which exploits additional information to facilitate
learning but does not require extra information in testing. There are a few baseline models falling into the same paradigm that we can compare with. One is the SVM+ method (Pechyony & Vapnik, 2011) and the other one is the standard model (i.e., the ResNet-18). We simply refer to ResNet-18 by baseline if no confusion occurs. In the experiments, we implement the SVM+ using the code provided by Pechyony & Vapnik (2011) with default parameter settings and linear kernel. We follow the scheme as described in Lapin et al. (2014) to train the SVM+ model. More concretely, we train multiple one-versus-rest SVM+ models upon the deep features extracted from both the entire images and the foreground regions (used as the privileged information). We use the averaged pooling over 10 crops on the feature maps before the FC layer as the deep feature for training SVM+. It is worth noting that all of these models (including SVM+ and GoCNN) use a linear classifier and thus have the same number of parameters, or more concretely, GoCNN does not require more parameters than SVM+ and the vanilla ResNet.
5.2 TRAINING MODELS WITH COMPLETE PRIVILEGED INFORMATION
In this subsection, we consider the scenario where every training sample has complete privileged segmentation information. Firstly, we evaluate the performance of our proposed GoCNN on the ImageNet-0.1m dataset. Table 1 summarizes the accuracy of different models. As can be seen from the results, given the complete privileged information, our proposed GoCNN presents much better performance than compared models. The group orthogonal constraints successfully regularize the learned feature to be within the foreground and background. The trained GoCNN thus presents a stronger generalization ability. It is also interesting (although not surprising) to observe that, when foreground features with background features are combined, the performance of GoCNN can be further improved from 49.60% to 50.39% in terms of top-1 accuracy. One can observe that the background information indeed benefits object recognition to some extent. To further investigate the contribution of each component within GoCNN to final performance, we conduct another experiment and show the results in Table 2. In the experiments, we purposively prevent the gradient propagation from the other components except the one being investigated during training, and perform another setting on the baseline method where the background is removed and only the foreground object is reserved in each training sample, noted as Baseline-obj. Comparing the result of Full GoCNN between different classifiers, we can see that learning background features can actually improve the overall performance. And when we compare the Fg classifier between Baseline-obj, Only Fg and Full GoCNN, we can see the importance of the background information in training more robust and richer foreground features.
Secondly, to verify the effectiveness of learning features in two different groups with our proposed method, we visualize the maximum activation value within each group of feature maps of several testing images. The feature maps are generated by the final convolution layer with 384 × 384 resolution input testing images. Then, the final convolution layer gives 12 × 12 output maps. We aggregate feature maps within the same group into one feature map by max operation. As can be seen from Figure 2, foreground and background features are well separated and the result looks just like the semantic segmentation mask. Compared with the baseline model, more neurons are activated in our proposed method in the two orthogonal spaces. This indicates that more diverse and discriminative features are learned in our framework compared with the baseline method. Finally, we further evaluate the generalization ability of our proposed method on the PASCAL VOC dataset. It is well known that an object shares many common properties with others even if they are not from the same category. A well-performing CNN model should be able to learn robust features rather than just fit the training images. In this experiment, we fine-tune different models on the PASCAL VOC images to test whether the learned features are able to generalize well to another dataset. Note that
we add another convolution layer with a 1 × 1 kernel size and 512 outputs as an adaptive layer on all models. It is not necessary to add such a layer in networks without a residual structure (He et al., 2015). As can be seen from Table 3, our proposed network shows better results and higher average precision across all categories, which means our proposed GoCNN learns more representative and richer features that are easier to transfer from one domain to another.
5.3 TRAINING GOCNN WITH PARTIAL PRIVILEGED INFORMATION
In this subsection, we investigate the performance of different models with only using partial privileged information. The experiment is also conducted on the ImageNet-0.1m dataset. We evaluate the performance of our proposed GoCNN by varying the percentage of privileged information (i.e., percentage of training images with segmentation annotations) from 20% to 100%.
The validation accuracies of GoCNN and the baseline model (i.e., the ResNet-18) are shown in Table 4. From the results, one can observe that with the increasing percentage of privileged information, the accuracy will continuously increase until the percentage of privileged information reaches 80%. The performance on increasing the percentage from 40% to 100% is only 0.71% compared with 0.92% on the increasing from 20% to 40%. This is probably because the suppression losses are more effective than we expected; that is, with very little guidance from the suppression
loss, the network can already be able to separate foreground and background features and explore new features within each group.
To verify the effectiveness of GoCNN on very large training dataset with more complex CNN architectures, we conducted another experiment on the complete ImageNet-1k dataset with only 10% privileged information, and we use the 152-layer ResNet as our basic model. As can be seen from Table 5, our proposed GoCNN achieves 21.8% top-1 error while the vanilla ResNet-152 has 23.0% top-1 error. Such performance boost is consistent with the results shown in Table 4, which again confirms the effectiveness of the GoCNN.
6 DISCUSSIONS
Based on our experimental results, we can also provide answers to the following two important questions.
Does background information indeed help object recognition for deep learning methods? Based on our experiments, we give a positive answer. Intuitively, background information may provide some “hints” for object recognition. However, though several works (Song et al., 2011; Russakovsky et al., 2012) have proven the usefulness of background information when using handcraft features, few works have studied the effectiveness of background information on deep learning methods for object recognition tasks. Based on the experimental results shown in Table 2, both the foreground classification accuracy and the overall classification accuracy can be further boosted with our proposed framework. This means that the background deep features can also provide useful information for foreground object recognition.
Can a more precise annotation with richer information, e.g., segmentation annotation, assist the image classification training process? The answer is clearly yes. In fact, in recent years, several works have explored how object detection and segmentation can benefit each other (Dai et al., 2015; Hariharan et al., 2014). However, none of existing works has studied how image segmentation information can help train a better classification deep neural network. In this work, by treating the segmentation annotations as the privileged information, we first demonstrate a possible way to utilize segmentation annotations to assist image classification training.
7 CONCLUSION
We proposed a group orthogonal neural network for image classification which encourages learning more diverse feature representations. Privileged information is utilized to train the proposed GoCNN model. To the best of our knowledge, we are the first to explore how to use image segmentation as privileged information to assist CNN training for image classification.

This paper proposes to learn groups of orthogonal features in a convnet by penalizing correlation among features in each group.  The technique is applied in the setting of image classification with “privileged information” in the form of foreground segmentation masks, where the model is trained to learn orthogonal groups of foreground and background features using the correlation penalty and an additional “background suppression” term.
Pros:
Proposes a “group-wise model diversity” loss term which is novel, to my knowledge.
The use of foreground segmentation masks to improve image classification is also novel.
The method is evaluated on two standard and relatively large-scale vision datasets: ImageNet and PASCAL VOC 2012.
Cons:
The evaluation is lacking.  There should be a baseline that leaves out the background suppression term, so readers know how much that term is contributing to the performance vs. the group orthogonal term.  The use of the background suppression term is also confusing to me -- it seems redundant, as the group orthogonality term should already serve to suppress the use of background features by the foreground feature extractor.
It would be nice to see the results with “Incomplete Privileged Information” on the full ImageNet dataset (rather than just 10% of it) with the privileged information included for the 10% of images where it’s available.  This would verify that the method and use of segmentation masks remains useful even in the regime of more labeled classification data.
The presentation overall is a bit confusing and difficult to follow, for me.  For example, Section 4.2 is titled “A Unified Architecture: GoCNN”, yet it is not an overview of the method as a whole, but a list of specific implementation details (even the very first sentence).
Minor: calling eq 3 a “regression loss” and writing “||0 - x||” rather than just “||x||” is not necessary and makes understanding more difficult -- I’ve never seen a norm regularization term written this way or described as a “regression to 0”.
Minor: in fig. 1 I think the FG and BG suppression labels are swapped: e.g., the “suppress foreground” mask has 1s in the FG and 0s in the BG (which would suppress the BG, not the FG).
An additional question: why are the results in Table 4 with 100% privileged information different from those in Table 1-2?  Are these not the same setting?
The ideas presented in this paper are novel and show some promise, but are currently not sufficiently ablated for readers to understand what aspects of the method are important.  Besides additional experiments, the paper could also use some reorganization and revision for clarity.
===============
Edit (1/29/17): after considering the latest revisions -- particularly the full ImageNet evaluation results reported in Table 5 demonstrating that the background segmentation 'privileged information' is beneficial even with the full labeled ImageNet dataset -- I've upgraded my rating from 4 to 6.
(I'll reiterate a very minor point about Figure 1 though: I still think the "0" and "1" labels in the top part of the figures should be swapped to match the other labels.  e.g., the topmost path in figure 1a, with the text "suppress foreground", currently has 0 in the background and 1 in the foreground, when one would want the reverse of this to suppress the foreground.)


Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on ε) require Ω(poly(1/ε)) neurons while deep networks (i.e., networks whose depth grows with 1/ε) require O(polylog(1/ε)) neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular type of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.
1 INTRODUCTION
Neural networks have drawn significant interest from the machine learning community, especially due to their recent empirical successes (see the surveys (Bengio, 2009)). Neural networks are used to build state-of-art systems in various applications such as image recognition, speech recognition, natural language process and others (see, Krizhevsky et al. 2012; Goodfellow et al. 2013; Wan et al. 2013, for example). The result that neural networks are universal approximators is one of the theoretical results most frequently cited to justify the use of neural networks in these applications. Numerous results have shown the universal approximation property of neural networks in approximations of different function classes, (see, e.g., Cybenko 1989; Hornik et al. 1989; Funahashi 1989; Hornik 1991; Chui & Li 1992; Barron 1993; Poggio et al. 2015).
All these results and many others provide upper bounds on the network size and assert that small approximation error can be achieved if the network size is sufficiently large. More recently, there has been much interest in understanding the approximation capabilities of deep versus shallow networks. Delalleau & Bengio (2011) have shown that there exist deep sum-product networks which cannot be approximated by shallow sum-product networks unless they use an exponentially larger amount of units or neurons. Montufar et al. (2014) have shown that the number of linear region increases exponentially with the number of layers in the neural network. Telgarsky (2016) has established such a result for neural networks, which is the subject of this paper. Eldan & Shamir (2015) have shown that, to approximate a specific function, a two-layer network requires an exponential number of neurons in the input dimension, while a three-layer network requires a polynomial number of neurons. These recent papers demonstrate the power of deep networks by showing that depth can lead to an exponential reduction in the number of neurons required, for specific functions or specific neural networks. Our goal here is different: we are interested in function approximation specifically
and would like to show that for a given upper bound on the approximation error, shallow networks require exponentially more neurons than deep networks for a large class of functions.
The multilayer neural networks considered in this paper are allowed to use either rectifier linear units (ReLU) or binary step units (BSU), or any combination of the two. The main contributions of this paper are
• We have shown that, for ε-approximation of functions with enough piecewise smoothness, a multilayer neural network which uses Θ(log(1/ε)) layers only needs O(poly log(1/ε)) neurons, while Ω(poly(1/ε)) neurons are required by neural networks with o(log(1/ε)) layers. In other words, shallow networks require exponentially more neurons than a deep network to achieve the level of accuracy for function approximation.
• We have shown that for all differentiable and strongly convex functions, multilayer neural networks need Ω(log(1/ε)) neurons to achieve an ε-approximation. Thus, our results for deep networks are tight.
The outline of this paper is as follows. In Section 2, we present necessary definitions and the problem statement. In Section 3, we present upper bounds on network size, while the lower bound is provided in Section 4. Conclusions are presented in Section 5. Around the same time that our paper was uploaded in arxiv, a similar paper was also uploaded in arXiv by Yarotsky (2016). The results in the two papers are similar in spirit, but the details and the general approach are substantially different.
2 PRELIMINARIES AND PROBLEM STATEMENT
In this section, we present definitions on feedforward neural networks and formally present the problem statement.
2.1 FEEDFORWARD NEURAL NETWORKS
A feedforward neural network is composed of layers of computational units and defines a unique function f̃ : Rd → R. Let L denote the number of hidden layers, Nl denote the number of units of layer l, N = ∑L l=1 Nl denote the size of the neural network, vector x = (x
(1), ..., x(d)) denote the input of neural network, zlj denote the output of the jth unit in layer l, w l i,j denote the weight of the edge connecting unit i in layer l and unit j in layer l + 1, blj denote the bias of the unit j in layer l. Then outputs between layers of the feedforward neural network can be characterized by following iterations:
zl+1j = σ ( Nl∑ i=1 wli,jz l i + b l+1 j ) , l ∈ [L− 1], j ∈ [Nl+1],
with
input layer: z1j = σ
( d∑
i=1
w0i,jx (i) + b1j ) , j ∈ [N1],
output layer: f̃(x) = σ ( NL∑ i=1 wLi,jz L i + b L+1 j ) .
Here, σ(·) denotes the activation function and [n] denotes the index set [n] = {1, ..., n}. In this paper, we only consider two important types of activation functions:
• Rectifier linear unit: σ(x) = max{0, x}, x ∈ R. • Binary step unit: σ(x) = I{x ≥ 0}, x ∈ R.
We call the number of layers and the number of neurons in the network as the depth and the size of the feedforward neural network, respectively. We use the set F(N,L) to denote the function set containing all feedforward neural networks of depth L, size N and composed of a combination
of rectifier linear units (ReLUs) and binary step units. We say one feedforward neural network is deeper than the other network if and only if it has a larger depth. Through this paper, the terms feedforward neural network and multilayer neural network are used interchangeably.
2.2 PROBLEM STATEMENT
In this paper, we focus on bounds on the size of the feedforward neural network function approximation. Given a function f , our goal is to understand whether a multilayer neural network f̃ of depth L and size N exists such that it solves
min f̃∈F(N,L)
∥f − f̃∥ ≤ ε. (1)
Specifically, we aim to answer the following questions:
1 Does there exists L(ε) and N(ε) such that (1) is satisfied? We will refer to such L(ε) and N(ε) as upper bounds on the depth and size of the required neural network.
2 Given a fixed depth L, what is the minimum value of N such that (1) is satisfied? We will refer to such an N as a lower bound on the size of a neural network of a given depth L.
The first question asks what depth and size are sufficient to guarantee an ε-approximation. The second question asks, for a fixed depth, what is the minimum size of a neural network required to guarantee an ε-approximation. Obviously, tight bounds in the answers to these two questions provide tight bounds on the network size and depth required for function approximation. Besides, solutions to these two questions together can be further used to answer the following question. If a deeper neural network of sizeNd and a shallower neural network of sizeNs are used to approximate the same function with the same error, then how fast does the ratioNd/Ns decay to zero as the error decays to zero?
3 UPPER BOUNDS ON FUNCTION APPROXIMATIONS
In this section, we present upper bounds on the size of the multilayer neural network which are sufficient for function approximation. Before stating the results, some notations and terminology deserve further explanation. First, the upper bound on the network size represents the number of neurons required at most for approximating a given function with a certain error. Secondly, the notion of the approximation is the L∞ distance: for two functions f and g, the L∞ distance between these two function is the maximum point-wise disagreement over the cube [0, 1]d.
3.1 APPROXIMATION OF UNIVARIATE FUNCTIONS
In this subsection, we present all results on approximating univariate functions. We first present a theorem on the size of the network for approximating a simple quadratic function. As part of the proof, we present the structure of the multilayer feedforward neural network used and show how the neural network parameters are chosen. Results on approximating general functions can be found in Theorem 2 and 4.
Theorem 1. For function f(x) = x2, x ∈ [0, 1], there exists a multilayer neural network f̃(x) with O ( log 1ε ) layers, O ( log 1ε ) binary step units and O ( log 1ε ) rectifier linear units such that |f(x)− f̃(x)| ≤ ε, ∀x ∈ [0, 1].
Proof. The proof is composed of three parts. For any x ∈ [0, 1], we first use the multilayer neural network to approximate x by its finite binary expansion ∑n i=0 xi 2i . We then construct a 2-layer neural
network to implement function f (∑n
i=0 xi 2i
) .
For each x ∈ [0, 1], x can be denoted by its binary expansion x = ∑∞
i=0 xi 2i , where xi ∈ {0, 1} for
all i ≥ 0. It is straightforward to see that the n-layer neural network shown in Figure 1 can be used to find x0, ..., xn.
Next, we implement the function f̃(x) = f (∑n
i=0 xi 2i
) by a two-layer neural network. Since
f(x) = x2, we then rewrite f̃(x) as follows:
f̃(x) =
( n∑
i=0
xi 2i
)2 =
n∑ i=0 xi ·  1 2i n∑ j=0 xj 2j  = n∑ i=0 max 0, 2(xi − 1) + 1 2i n∑ j=0 xj 2j  . The third equality follows from the fact that xi ∈ {0, 1} for all i. Therefore, the function f̃(x) can be implemented by a multilayer network containing a deep structure shown in Figure 1 and another hidden layer with n rectifier linear units. This multilayer neural network has O(n) layers, O(n) binary step units and O(n) rectifier linear units. Finally, we consider the approximation error of this multilayer neural network,
|f(x)− f̃(x)| = ∣∣∣∣∣∣x2 − ( n∑ i=0 xi 2i )2∣∣∣∣∣∣ ≤ 2 ∣∣∣∣∣x− n∑ i=0 xi 2i ∣∣∣∣∣ = 2 ∣∣∣∣∣ ∞∑ i=n+1 xi 2i ∣∣∣∣∣ ≤ 12n−1 . Therefore, in order to achieve ε-approximation error, one should choose n = ⌈ log2 1 ε ⌉ + 1. In
summary, the deep neural network hasO ( log 1ε ) layers,O ( log 1ε ) binary step units andO ( log ( 1 ε )) rectifier linear units.
Next, a theorem on the size of the network for approximating general polynomials is given as follows. Theorem 2. For polynomials f(x) = ∑p i=0 aix i, x ∈ [0, 1] and ∑p
i=1 |ai| ≤ 1, there exists a multilayer neural network f̃(x) withO ( p+ log pε ) layers, O ( log pε ) binary step units andO ( p log pε ) rectifier linear units such that |f(x)− f̃(x)| ≤ ε, ∀x ∈ [0, 1].
Proof. The proof is composed of three parts. We first use the deep structure shown in Figure 1 to find the n-bit binary expansion ∑n i=0 aix
i of x. Then we construct a multilayer network to approximate polynomials gi(x) = xi, i = 1, ..., p. Finally, we analyze the approximation error.
Using the same deep structure shown in Figure 1, we could find the binary expansion sequence {x0, ..., xn}. In this step, we used n binary steps units in total. Now we rewrite gm+1( ∑n i=0 xi 2n ),
gm+1
( n∑
i=0
xi 2i
) =
n∑ j=0
[ xj · 1
2j gm
( n∑
i=0
xi 2i
)] =
n∑ j=0 max
[ 2(xj − 1) + 1
2j gm
( n∑
i=0
xi 2i
) , 0 ] .
(2) Clearly, the equation (2) defines iterations between the outputs of neighbor layers. Therefore, the deep neural network shown in Figure 2 can be used to implement the iteration given by (2). Further, to implement this network, one should useO(p) layers withO(pn) rectifier linear units in total. We now define the output of the multilayer neural network as f̃(x) = ∑p i=0 aigi (∑n j=0 xj 2j ) . For this multilayer network, the approximation error is
|f(x)− f̃(x)| = ∣∣∣∣∣∣ p∑
i=0
aigi  n∑ j=0 xj 2j − p∑ i=0 aix i ∣∣∣∣∣∣ ≤ p∑ i=0 |ai| · ∣∣∣∣∣∣gi  n∑ j=0 xj 2j − xi ∣∣∣∣∣∣  ≤ p 2n−1
This indicates, to achieve ε-approximation error, one should choose n = ⌈ log pε ⌉ + 1. Besides, since we used O(n + p) layers with O(n) binary step units and O(pn) rectifier linear units in total, this multilayer neural network thus has O ( p+ log pε ) layers, O ( log pε ) binary step units and
O ( p log pε ) rectifier linear units.
In Theorem 2, we have shown an upper bound on the size of multilayer neural network for approximating polynomials. We can easily observe that the number of neurons in network grows as p log p with respect to p, the degree of the polynomial. We note that both Andoni et al. (2014) and Barron (1993) showed the sizes of the networks grow exponentially with respect to p if only 3-layer neural networks are allowed to be used in approximating polynomials.
Besides, every function f with p + 1 continuous derivatives on a bounded set can be approximated easily with a polynomial with degree p. This is shown by the following well known result of Lagrangian interpolation. By this result, we could further generalize Theorem 2. The proof can be found in the reference (Gil et al., 2007). Lemma 3 (Lagrangian interpolation at Chebyshev points). If a function f is defined at points z0, ..., zn, zi = cos((i+ 1/2)π/(n+ 1)), i ∈ [n], there exists a polynomial of degree not more than n such that Pn(zi) = f(zi), i = 0, ..., n. This polynomial is given by Pn(x) = ∑n i=0 f(zi)Li(x) where Li(x) = πn+1(x)
(x−zi)π′n+1(zi) and πn+1(x) =
∏n j=0(x − zj). Additionally, if f is continuous on
[−1, 1] and n+ 1 times differentiable in (−1, 1), then
∥Rn∥ = ∥f − Pn∥ ≤ 1
2n(n+ 1)! ∥∥∥f (n+1)∥∥∥ , where f (n)(x) is the derivative of f of the nth order and the norm ∥f∥ is the l∞ norm ∥f∥ = maxx∈[−1,1] f(x).
Then the upper bound on the network size for approximating more general functions follows directly from Theorem 2 and Lemma 3. Theorem 4. Assume that function f is continuous on [0, 1] and ⌈ log 2ε ⌉ + 1 times differentiable in
(0, 1). Let f (n) denote the derivative of f of nth order and ∥f∥ = maxx∈[0,1] f(x). If ∥∥f (n)∥∥ ≤ n!
holds for all n ∈ [⌈ log 2ε ⌉ + 1 ] , then there exists a deep neural network f̃ with O ( log 1ε ) layers,
O ( log 1ε ) binary step units, O (( log 1ε )2) rectifier linear units such that ∥∥∥f − f̃∥∥∥ ≤ ε. Proof. Let N = ⌈ log 2ε ⌉ . From Lemma 3, it follows that there exists polynomial PN of degree N such that for any x ∈ [0, 1],
|f(x)− PN (x)| ≤ ∥∥f (N+1)∥∥ 2N (N + 1)! ≤ 1 2N .
Let x0, ..., xN denote the first N + 1 bits of the binary expansion of x and define f̃(x) = PN (∑N i=0 xi 2N ) . In the following, we first analyze the approximation error of f̃ and next
show the implementation of this function. Let x̃ = ∑N
i=0 xi 2i . The error can now be upper bounded
by
|f(x)− f̃(x)| = |f(x)− PN (x̃)| ≤ |f(x)− f (x̃)|+ |f (x̃)− PN (x̃)| ≤ ∥∥∥f (1)∥∥∥ · ∣∣∣∣∣x− N∑ i=0 xi 2i ∣∣∣∣∣+ 12N ≤ 12N + 12N ≤ ε In the following, we describe the implementation of f̃ by a multilayer neural network. Since PN is a polynomial of degree N , function f̃ can be rewritten as
f̃(x) = PN ( N∑ i=0 xi 2i ) = N∑ n=0 cngn ( N∑ i=0 xi 2i ) for some coefficients c0, ..., cN and gn = xn, n ∈ [N ]. Hence, the multilayer neural network shown in the Figure 2 can be used to implement f̃(x). Notice that the network usesO(N) layers withO(N) binary step units in total to decode x0,...,xN and O(N) layers with O(N2) rectifier linear units in total to construct the polynomial PN . Substituting N = ⌈ log 2ε ⌉ , we have proved the theorem.
Remark: Note that, to implement the architecture in Figure 2 using the definition of a feedforward neural network in Section 2, we need the gi, i ∈ [p] at the output. This can be accomplished by using O(p2) additional ReLUs. Since p = O(log(1/ε)), this doesn’t change the order result in Theorem 4.
Theorem 4 shows that any function f with enough smoothness can be approximated by a multilayer neural network containing polylog ( 1 ε ) neurons with ε error. Further, Theorem 4 can be used to show that for functions h1,...,hk with enough smoothness, then linear combinations, multiplications and compositions of these functions can as well be approximated by multilayer neural networks containing polylog ( 1 ε ) neurons with ε error. Specific results are given in the following corollaries. Corollary 5 (Function addition). Suppose that all functions h1, ..., hk satisfy the conditions in Theorem 4, and the vector β ∈ {ω ∈ Rk : ∥ω∥1 = 1}, then for the linear combination f = ∑k i=1 βihi, there exists a deep neural network f̃ with O ( log 1ε ) layers, O ( log 1ε ) binary
step units, O (( log 1ε )2) rectifier linear units such that |f(x)− f̃ | ≤ ε, ∀x ∈ [0, 1].
Remark: Clearly, Corollary 5 follows directly from the fact that the linear combination f satisfies the conditions in Theorem 4 if all the functions h1,...,hk satisfy those conditions. We note here that the upper bound on the network size for approximating linear combinations is independent of k, the number of component functions. Corollary 6 (Function multiplication). Suppose that all functions h1,...,hk are continuous on [0, 1] and ⌈ 4k log2 4k + 4k + 2 log2 2 ε ⌉ + 1 times differentiable in (0, 1). If ∥h(n)i ∥ ≤ n! holds for all
i ∈ [k] and n ∈ [⌈ 4k log2 4k + 4k + 2 log2 2 ε ⌉ + 1 ] then for the multiplication f = ∏k i=1 hi,
there exists a multilayer neural network f̃ with O ( k log k + log 1ε ) layers, O ( k log k + log 1ε ) bi-
nary step units and O ( (k log k)2 + ( log 1ε )2) rectifier linear units such that |f(x) − f̃(x)| ≤ ε, ∀x ∈ [0, 1]. Corollary 7 (Function composition). Suppose that all functions h1, ..., hk : [0, 1] → [0, 1] satisfy the conditions in Theorem 4, then for the composition f = h1 ◦ h2 ◦ ... ◦ hk, there exists a multilayer neural network f̃ with O ( k log k log 1ε + log k ( log 1ε )2) layers,
O ( k log k log 1ε + log k ( log 1ε )2) binary step units and O ( k2 ( log 1ε )2 + ( log 1ε )4) rectifier lin-
ear units such that |f(x)− f̃(x)| ≤ ε, ∀x ∈ [0, 1].
Remark: Proofs of Corollary 6 and 7 can be found in the appendix. We observe that different from the case of linear combinations, the upper bound on the network size grows as k2 log2 k in the case of function multiplications and grows as k2 ( log 1ε )2 in the case of function compositions where k is the number of component functions.
In this subsection, we have shown a polylog ( 1 ε ) upper bound on the network size for εapproximation of both univariate polynomials and general univariate functions with enough smoothness. Besides, we have shown that linear combinations, multiplications and compositions of univariate functions with enough smoothness can as well be approximated with ε error by a multilayer neural network of size polylog ( 1 ε ) . In the next subsection, we will show the upper bound on the network size for approximating multivariate functions.
3.2 APPROXIMATION OF MULTIVARIATE FUNCTIONS
In this subsection, we present all results on approximating multivariate functions. We first present a theorem on the upper bound on the neural network size for approximating a product of multivariate linear functions. We next present a theorem on the upper bound on the neural network size for approximating general multivariate polynomial functions. Finally, similar to the results in the univariate case, we present the upper bound on the neural network size for approximating the linear combination, the multiplication and the composition of multivariate functions with enough smoothness. Theorem 8. Let W = {w ∈ Rd : ∥w∥1 = 1}. For f(x) = ∏p i=1 ( wTi x ) , x ∈ [0, 1]d and
wi ∈ W , i = 1, ..., p, there exists a deep neural network f̃(x) with O ( p+ log pdε ) layers and
O ( log pdε ) binary step units and O ( pd log pdε ) rectifier linear units such that |f(x)− f̃(x)| ≤ ε, ∀x ∈ [0, 1]d.
Theorem 8 shows an upper bound on the network size for ε-approximation of a product of multivariate linear functions. Furthermore, since any general multivariate polynomial can be viewed as a linear combination of products, the result on general multivariate polynomials directly follows from Theorem 8. Theorem 9. Let the multi-index vectorα = (α1, ..., αd), the norm |α| = α1+...+αd, the coefficient
Cα = Cα1...αd , the input vector x = (x
(1), ..., x(d)) and the multinomial xα = x(1) α1 ...x(d)
αd . For positive integer p and polynomial f(x) = ∑ α:|α|≤p Cαx α, x ∈ [0, 1]d and ∑ α:|α|≤p |Cα| ≤ 1,
there exists a deep neural network f̃(x) of depth O ( p+ log dpε ) and size N(d, p, ε) such that |f(x)− f(x̃)| ≤ ε, where
N(d, p, ε) = p2 ( p+ d− 1 d− 1 ) log pd ε .
Remark: The proof is given in the appendix. By further analyzing the results on the network size, we obtain the following results: (a) fixing degree p, N(d, ε) = O ( dp+1 log dε ) as d → ∞ and
(b) fixing input dimension d, N(p, ε) = O ( pd log pε ) as p → ∞. Similar results on approximating multivariate polynomials were obtained by Andoni et al. (2014) and Barron (1993). Barron (1993) showed that on can use a 3-layer neural network to approximate any multivariate polynomial with degree p, dimension d and network size dp/ε2. Andoni et al. (2014) showed that one could use the gradient descent to train a 3-layer neural network of size d2p/ε2 to approximate any multivariate polynomial. However, Theorem 9 shows that the deep neural network could reduce the network size from O (1/ε) to O ( log 1ε ) for the same ε error. Besides, for a fixed input dimension d, the size of the 3-layer neural network used by Andoni et al. (2014) and Barron (1993) grows exponentially with respect to the degree p. However, the size of the deep neural network shown in Theorem 9 grows only polynomially with respect to the degree. Therefore, the deep neural network could reduce the network size from O(exp(p)) to O(poly(p)) when the degree p becomes large. Theorem 9 shows an upper bound on the network size for approximating multivariate polynomials. Further, by combining Theorem 4 and Corollary 7, we could obtain an upper bound on the network size for approximating more general functions. The results are shown in the following corollary. Corollary 10. Assume that all univariate functions h1, ..., hk : [0, 1] → [0, 1], k ≥ 1, satisfy the conditions in Theorem 4. Assume that the multivariate polynomial l(x) : [0, 1]d → [0, 1] is of degree p. For composition f = h1 ◦ h2 ◦ ... ◦ hk ◦ l(x), there exists a multilayer neural network f̃ of depth O ( p+ log d+ k log k log 1ε + log k ( log 1ε )2) and of sizeN(k, p, d, ε) such that |f̃(x)−f(x)| ≤ ε for ∀x ∈ [0, 1]d, where
N(k, p, d, ε) = O ( p2 ( p+ d− 1 d− 1 ) log pd ε + k2 ( log 1 ε )2 + ( log 1 ε )4) .
Remark: Corollary 10 shows an upper bound on network size for approximating compositions of multivariate polynomials and general univariate functions. The upper bound can be loose due to the assumption that l(x) is a general multivariate polynomials of degree p. For some specific cases, the upper bound can be much smaller. We present two specific examples in the Appendix H and I.
In this subsection, we have shown that a similar polylog ( 1 ε ) upper bound on the network size for ε-approximation of general multivariate polynomials and functions which are compositions of univariate functions and multivariate polynomials.
The results in this section can be used to find a multilayer neural network of size polylog ( 1 ε ) which provides an approximation error of at most ε. In the next section, we will present lower bounds on the network size for approximating both univariate and multivariate functions. The lower bound together with the upper bound shows a tight bound on the network size required for function approximations.
While we have presented results in both the univariate and multivariate cases for smooth functions, the results automatically extend to functions that are piecewise smooth, with a finite number of pieces. In other words, if the domain of the function is partitioned into regions, and the function is sufficiently smooth (in the sense described in the Theorems and Corollaries earlier) in each of the regions, then the results essentially remain unchanged except for an additional factor which will depend on the number of regions in the domain.
4 LOWER BOUNDS ON FUNCTION APPROXIMATIONS
In this section, we present lower bounds on the network size in function for certain classes of functions. Next, by combining the lower bounds and the upper bounds shown in the previous section, we could analytically show the advantages of deeper neural networks over shallower ones. The theorem below is inspired by a similar result (DasGupta & Schnitger, 1993) for univariate quadratic functions, where it is stated without a proof. Here we show that the result extends to general multivariate strongly convex functions. Theorem 11. Assume function f : [0, 1]d → R is differentiable and strongly convex with parameter µ. Assume the multilayer neural network f̃ is composed of rectifier linear units and binary step units. If |f(x)− f̃(x)| ≤ ε, ∀x ∈ [0, 1]d, then the network size N ≥ log2 ( µ 16ε ) .
Remark: The proof is in the Appendix F. Theorem 11 shows that every strongly convex function cannot be approximated with error ε by any multilayer neural network with rectifier linear units and binary step units and of size smaller than log2(µ/ε) − 4. Theorem 11 together with Theorem 1 directly shows that to approximate quadratic function f(x) = x2 with error ε, the network size should be of order Θ ( log 1ε ) . Further, by combining Theorem 11 and Theorem 4, we could analytically show the benefits of deeper neural networks. The result is given in the following corollary. Corollary 12. Assume that univariate function f satisfies conditions in both Theorem 4 and Theorem 11. If a neural network f̃s is of depth Ls = o ( log 1ε ) , size Ns and |f(x) − f̃s(x)| ≤ ε,
for ∀x ∈ [0, 1], then there exists a deeper neural network f̃d(x) of depth Θ ( log 1ε ) , size Nd = O(L2s log 2 Ns) such that |f(x)− f̃d(x)| ≤ ε, ∀x ∈ [0, 1].
Remarks: (i) The strong convexity requirement can be relaxed: the result obviously holds if the function is strongly concave and it also holds if the function consists of pieces which are strongly convex or strongly concave. (ii) Corollary 12 shows that in the approximation of the same function, the size of the deep neural network Ns is only of polynomially logarithmic order of the size of the shallow neural network Nd, i.e., Nd = O(polylog(Ns)). Similar results can be obtained for multivariate functions on the type considered in Section 3.2.
5 CONCLUSIONS
In this paper, we have shown that an exponentially large number of neurons are needed for function approximation using shallow networks, when compared to deep networks. The results are established for a large class of smooth univariate and multivariate functions. Our results are established for the case of feedforward neural networks with ReLUs and binary step units.
ACKNOWLEDGMENTS
The research reported here was supported by NSF Grants CIF 14-09106, ECCS 16-09370, and ARO Grant W911NF-16-1-0259.
APPENDIX A PROOF OF COROLLARY 5
Proof. By Theorem 4, for each hi, i = 1, ..., k, there exists a multilayer neural network h̃i such that |hi(x)− h̃(x)| ≤ ε for any x ∈ [0, 1]. Let
f̃(x) = k∑ i=1 βih̃i(x).
Then the approximation error is upper bounded by
|f(x)− f̃(x)| = ∣∣∣∣∣ k∑
i=1
βihi(x) ∣∣∣∣∣ ≤ k∑
i=1
|βi| · |hi(x)− h̃(x)| = ε.
Now we compute the size of the multilayer neural network f̃ . LetN = ⌈ log 2ε ⌉ and ∑N i=0 xi 2i be the binary expansion of x. Since h̃i(x) has a form of
h̃i(x) = N∑ j=0 cijgj ( N∑ i=0 xi 2i ) ,
where gj(x) = xj , then f̃ should has a form of
f̃(x) = k∑ i=1 βi  N∑ j=0 cijgj ( N∑ i=0 xi 2i ) and can be further rewritten as
f̃(x) = N∑ j=0
[( k∑
i=1
cijβi ) · gj ( N∑ i=0 xi 2i )] ≜ N∑ j=0 c′jgj ( N∑ i=0 xi 2i ) ,
where c′j = ∑ i cijβi. Therefore, f̃ can be implemented by a multilayer neural network shown in
Figure 2 and this network has at most O ( log 1ε ) layers, O ( log 1ε ) binary step units, O (( log 1ε )2) rectifier linear units.
APPENDIX B PROOF OF COROLLARY 6
Proof. Since f(x) = h1(x)h2(x)...hk(x), then the derivative of f of order n is
f (n) = ∑
α1+...+αk=n α1≥0,...,αk≥0
n!
α1!α2!...αk! h (α1) 1 h (α2) 2 ...h (αk) k .
By the assumption that ∥∥∥h(αi)i ∥∥∥ ≤ αi! holds for i = 1, ..., k, then we have∥∥∥f (n)∥∥∥ ≤ ∑
α1+...+αk=n α1≥0,...,αk≥0
n!
α1!α2!...αk!
∥∥∥h(α1)1 h(α2)2 ...h(αk)k ∥∥∥ ≤ (n+ k − 1k − 1 ) n!.
Then from Theorem 4, it follows that there exists a polynomial of PN degree N that ∥RN∥ = ∥f − PN∥ ≤ ∥∥f (N+1)∥∥ (N + 1)!2N ≤ 1 2N ( N + k k − 1 ) .
Since( N + k k − 1 ) ≤ (N + k) N+k (k − 1)k−1(N + 1)N+1 = ( N + k k − 1 )k−1 ( 1 + k − 1 N + 1 )N+1 ≤ ( e(N + k) k − 1 )k−1
then the error has an upper bound of
∥RN∥ ≤ (eN)k
2N ≤ 22k+k log2 N−N . (3)
Since we need to bound ∥RN∥ ≤ ε
2 ,
then we need to choose N such that
N ≥ k log2 N + 2k + log2 2
ε .
Thus, N can be chosen such that
N ≥ 2k log2 N and N ≥ 4k + 2 log2 2
ε .
Further, function l(x) = x/ log2 x is monotonically increasing on [e,∞) and
l(4k log2 4k) = 4k log2 4k log2 4k + log2 log2 4k ≥ 4k log2 4k log2 4k + log2 4k = 2k.
Therefore, to suffice the inequality (3), one should should choose
N ≥ 4k log2 4k + 4k + 2 log2 2
ε . Since N = ⌈ 4k log2 4k + 4k + 2 log2 2 ε ⌉ by assumptions, then there exists a polynomial PN of degree N such that ∥f − PN∥ ≤ ε
2 . Let ∑N
i=0 xi 2i denote the binary expansion of x and let
f̃(x) = PN ( N∑ i=0 xi 2i ) .
The approximation error is
|f̃(x)− f(x)| ≤ ∣∣∣∣∣f(x)− f ( N∑ i=0 xi 2i )∣∣∣∣∣+ ∣∣∣∣∣f ( N∑ i=0 xi 2i ) − PN ( N∑ i=0 xi 2i )∣∣∣∣∣ ≤ ∥f(1)∥
∣∣∣∣∣x− N∑ i=0 xi 2i ∣∣∣∣∣+ ε2 ≤ ε Further, function f̃ can be implemented by a multilayer neural network shown in Figure 2 and this network has at most O(N) layers, O(N) binary step units and O(N2) rectifier linear units.
APPENDIX C PROOF OF COROLLARY 7
Proof. We prove this theorem by induction. Define function Fm = h1 ◦ ... ◦ hm, m = 1, ..., k. Let T1(m) log3 3m ε , T2(m) log3 3m ε and T3(m) ( log3 3m ε )2 denote the number of layers, the number of binary step units and the number of rectifier linear units required at most for ε-approximation of Fm, respectively. By Theorem 4, for m = 1, there exists a multilayer neural network F̃1 with at most T1(1) log3 3 ε layers, T2(1) log3 3 ε binary step units and T3(1) ( log3 3 ε )2 rectifier linear units such that |F1(x)− F̃1(x)| ≤ ε, for x ∈ [0, 1]. Now we consider the cases for 2 ≤ m ≤ k. We assume for Fm−1, there exists a multilayer neural network F̃m−1 with not more than T1(m− 1) log3 3 m ε layers, T2(m− 1) log3 3m ε binary step units
and T3(m− 1) ( log3 3m
ε
)2 rectifier linear units such that
|Fm−1(x)− F̃m−1(x)| ≤ ε
3 , for x ∈ [0, 1].
Further we assume the derivative of Fm−1 has an upper bound ∥∥F ′m−1∥∥ ≤ 1. Then for Fm, since Fm(x) can be rewritten as Fm(x) = Fm−1(hm(x)), and there exists a multilayer neural network h̃m with at most T1(1) log3 3 ε layers, T2(1) log3 3 ε binary
step units and T3(1) ( log3 3 ε )2 rectifier linear units such that
|hm(x)− h̃m(x)| ≤ ε
3 , for x ∈ [0, 1], and ∥∥∥h̃m∥∥∥ ≤ (1 + ε/3). Then for cascaded multilayer neural network F̃m = F̃m−1 ◦ ( 11+ε/3 h̃m), we have
∥ Fm − F̃m ∥ = ∥∥∥∥∥Fm−1(hm)− F̃m−1 ( h̃m 1 + ε/3 )∥∥∥∥∥ ≤ ∥∥∥∥∥Fm−1(hm)− Fm−1 ( h̃m 1 + ε/3 )∥∥∥∥∥+ ∥∥∥∥∥Fm−1 ( h̃m 1 + ε/3 ) − F̃m−1 ( h̃m 1 + ε/3
)∥∥∥∥∥ ≤ ∥∥F ′m−1∥∥ · ∥∥∥∥∥hm − h̃m1 + ε/3
∥∥∥∥∥+ ε3 ≤ ∥∥F ′m−1∥∥ · ∥∥∥hm − h̃m∥∥∥+ ∥∥F ′m−1∥∥ · ∥∥∥∥ ε/31 + ε/3 h̃m ∥∥∥∥+ ε3
≤ ε 3 + ε 3 + ε 3 = ε
In addition, the derivative of Fm can be upper bounded by ∥F ′m∥ ≤ ∥∥F ′m−1∥∥ · ∥h′m∥ = 1. Since the multilayer neural network F̃m is constructed by cascading multilayer neural networks F̃m−1 and h̃m, then the iterations for T1, T2 and T3 are
T1(m) log3 3m
ε =T1(m− 1) log3
3m
ε + T1(1) log3
3 ε , (4)
T2(m) log3 3m
ε =T2(m− 1) log3
3m
ε + T2(1) log3
3 ε , (5)
T3(m) ( log3 3m
ε
)2 =T3(m− 1) ( log3 3m
ε
)2 + T3(1) ( log3 3
ε
)2 . (6)
From iterations (4) and (5), we could have for 2 ≤ m ≤ k,
T1(m) = T1(m− 1) + T1(1) 1 + log3(1/ε)
m+ log3(1/ε) ≤ T1(m− 1) + T1(1)
1 + log3(1/ε)
m
T2(m) = T2(m− 1) + T2(1) 1 + log3(1/ε)
m+ log3(1/ε) ≤ T2(m− 1) + T2(1)
1 + log3(1/ε)
m
and thus
T1(k) = O ( log k log 1
ε
) , T2(k) = O ( log k log 1
ε
) .
From the iteration (6), we have for 2 ≤ m ≤ k, T3(m) = T3(m− 1) + T3(1) ( 1 + log3(1/ε)
m+ log3(1/ε)
)2 ≤ T3(m− 1) + (1 + log3(1/ε)) 3
m2 ,
and thus
T3(k) = O (( log 1
ε
)2) .
Therefore, to approximate f = Fk, we need at most O ( k log k log 1ε + log k ( log 1ε )2) layers,
O ( k log k log 1ε + log k ( log 1ε )2) binary step units and O ( k2 ( log 1ε )2 + ( log 1ε )4) rectifier linear units.
APPENDIX D PROOF OF THEOREM 8
Proof. The proof is composed of two parts. As before, we first use the deep structure shown in Figure 1 to find the binary expansion of x and next use a multilayer neural network to approximate the polynomial.
Let x = (x(1), ..., x(d)) and wi = (wi1, ..., wid). We could now use the deep structure shown in Figure 1 to find the binary expansion for each x(k), k ∈ [d]. Let x̃(k) = ∑n
r=0 x(k)r 2r denote the binary
expansion of x(k), where x(k)r is the rth bit in the binary expansion of x(k). Obviously, to decode all the n-bit binary expansions of all x(k), k ∈ [d], we need a multilayer neural network with n layers and dn binary units in total. Besides, we let x̃ = (x̃(1), ..., x̃(d)). Now we define
f̃(x) = f(x̃) = p∏ i=1
( d∑
k=1
wikx̃ (k) ) .
We further define
gl(x̃) = l∏ i=1
( d∑
k=1
wikx̃ (k) ) .
Since for l = 1, ..., p− 1,
gl(x̃) = l∏ i=1
( d∑
k=1
wikx̃ (k) ) ≤
l∏ i=1 ∥wi∥1 = 1,
then we can rewrite gl+1(x̃), l = 1, ..., p− 1 into
gl+1(x̃) = l+1∏ i=1
( d∑
k=1
wikx̃ (k) ) =
d∑ k=1 [ w(l+1)kx̃ (k) · gl(x̃) ] = d∑ k=1 { w(l+1)k n∑ r=0 [ x(k)r · gl(x̃) 2r ]}
= d∑ k=1
{ w(l+1)k
n∑ r=0 max [ 2(x(k)r − 1) + gl(x̃) 2r , 0 ]} (7)
Obviously, equation (7) defines a relationship between the outputs of neighbor layers and thus can be used to implement the multilayer neural network. In this implementation, we need dn rectifier linear units in each layer and thus dnp rectifier linear units. Therefore, to implement function f̃(x), we need p+ n layers, dn binary step units and dnp rectifier linear units in total.
In the rest of proof, we consider the approximation error. Since for k = 1, ..., d and ∀x ∈ [0, 1]d,∣∣∣∣∂f(x)∂x(k) ∣∣∣∣ = ∣∣∣∣∣∣ p∑
j=1 wjk · p∏ i=1,i̸=j ( wTi x )∣∣∣∣∣∣ ≤ p∑ j=1 |wjk| ≤ p,
then
|f(x)− f̃(x)| = |f(x)− f(x̃)| ≤ ∥∇f∥2 · ∥x− x̃∥2 ≤ pd
2n .
By choosing n = ⌈ log2 pd ε ⌉ , we have
|f(x)− f(x̃)| ≤ ε.
Since we use nd binary step units to convert the input to binary form and dnp neurons in function approximation, we thus use O ( d log pdε ) binary step units and O ( pd log pdε ) rectifier linear units in total. In addition, since we have used n layers to convert the input to binary form and p layers in the function approximation section of the network, the whole deep structure has O ( p+ log pdε
) layers.
APPENDIX E PROOF OF THEOREM 9
Proof. For each multinomial function g with multi-index α, gα(x) = xα, it follows from Theorem 4 that there exists a deep neural network g̃α of size O ( |α| log |α|dε ) and depth
O ( |α|+ log |α|dε ) such that |gα(x)− g̃α(x)| ≤ ε. Let the deep neural network be
f̃(x) = ∑
α:|α|≤p
Cαg̃α(x),
and thus |f(x)− f̃(x)| ≤ ∑ α:|α|≤p |Cα| · |gα(x)− g̃α(x)| = ε.
Since the total number of multinomial is upper bounded by
p ( p+ d− 1 d− 1 ) ,
the size of deep neural network is thus upper bounded by p2 ( p+ d− 1 d− 1 ) log pd ε . (8)
If the dimension of the input d is fixed, then (8) is has the order of p2 ( p+ d− 1 d− 1 ) log pd ε = O ( (ep) d+1 log pd ε ) , p → ∞
while if the degree p is fixed, then (8) is has the order of p2 ( p+ d− 1 d− 1 ) log pd ε = O ( p2 (ed) p log pd ε ) , d → ∞.
APPENDIX F PROOF OF THEOREM 11
Proof. We first prove the univariate case d = 1. The proof is composed of two parts. We say the function g(x) has a break point at x = z if g is discontinuous at z or its derivative g′ is discontinuous at z. We first present the lower bound on the number of break pointsM(ε) that the multilayer neural network f̃ should have for ε-approximation of function f with error ε. We next relate the number of break points M(ε) to the network depth L and the size N .
Now we calculate the lower bound on M(ε). We first define 4 points x0, x1 = x0 + 2 √
ρε/µ, x2 = x1 + 2 √ ρε/µ and x3 = x2 + 2 √ ρε/µ, ∀ρ > 1. We assume
0 ≤ x0 < x1 < x2 < x3 ≤ 1.
We now prove that if multilayer neural network f̃ has no break point in [x1, x2], then f̃ should have a break point in [x0, x1] and a break point in [x2, x3]. We prove this by contradiction. We assume the neural network f̃ has no break points in the interval [x0, x3]. Since f̃ is constructed by rectifier linear units and binary step units and has no break points in the interval [x0, x3], then f̃ should be a linear function in the interval [x0, x3], i.e., f̃(x) = ax + b, x ∈ [x0, x3] for some a and b. By assumption, since f̃ approximates f with error at most ε everywhere in [0, 1], then
|f(x1)− ax1 − b| ≤ ε and |f(x2)− ax2 − b| ≤ ε.
Then we have f(x2)− f(x1)− 2ε
x2 − x1 ≤ a ≤ f(x2)− f(x1) + 2ε x2 − x1 .
By strong convexity of f , f(x2)− f(x1)
x2 − x1 +
µ 2 (x2 − x1) ≤ f ′(x2).
Besides, since ρ > 1 and µ
2 (x2 − x1) =
√ ρµε = 2ρε
x2 − x1 >
2ε
x2 − x1 ,
then a ≤ f ′(x2). (9) Similarly, we can obtain a ≥ f ′(x1). By our assumption that f̃ = ax+ b, x ∈ [x0, x3], then f(x3)− f̃(x3) = f(x3)− ax3 − b
= f(x3)− f(x2)− a(x3 − x2) + f(x2)− ax2 − b ≥ f ′(x2)(x3 − x2) + µ
2 (x3 − x2)2 − a(x3 − x2)− ε
= (f ′(x2)− a)(x3 − x2) + µ
2
( 2 √ ρε/µ )2 − ε
≥ (2ρ− 1)ε > ε The first inequality follows from strong convexity of f and f(x2) − ax2 − b ≥ ε. The second inequality follows from the inequality (9). Therefore, this leads to the contradiction. Thus there exists a break point in the interval [x2, x3]. Similarly, we could prove there exists a break point in the interval [x0, x1]. These indicate that to achieve ε-approximation in [0, 1], the multilayer neural network f̃ should have at least ⌈ 1 4 √ µ ρε ⌉ break points in [0, 1]. Therefore,
M(ε) ≥ ⌈ 1
4
√ µ
ρε
⌉ , ∀ρ > 1.
Further, Telgarsky (2016) has shown that the maximum number of break points that a multilayer neural network of depth L and size N could have is (N/L)L. Thus, L and N should satisfy
(N/L)L >
⌈ 1
4
√ µ
ρε
⌉ , ∀ρ > 1.
Therefore, we have
N ≥ L ( µ 16ε ) 1 2L .
Besides, letm = N/L. Since each layer in network should have at least 2 neurons, i.e.,m ≥ 2, then
N ≥ m 2 log2 m log2 ( µ 16ε ) ≥ log2 ( µ 16ε ) .
Now we consider the multivariate case d > 1. Assume input vector to be x = (x1, ..., x(d)). We now fix x(2), ..., x(d) and define two univariate functions
g(y) = f(y, x(2), ..., x(d)), and g̃(y) = f̃(y, x(2), ..., x(d)). By assumption, g(y) is a strongly convex function with parameter µ and for all y ∈ [0, 1], |g(y)− g̃(y)| ≤ ε. Therefore, by results in the univariate case, we should have
N ≥ L ( µ 16ε ) 1 2L and N ≥ log2 ( µ 16ε ) . (10)
Now we have proved the theorem.
Remark: We make the following remarks about the lower bound in the theorem.
(1) if the depth L is fixed, as in shallow networks, the number of neurons required is Ω ( (1/ε) 1 2L ) .
(2) if we are allowed to choose L optimally to minimize the lower bound, we will choose L = 12 log( µ 16ε ) and thus the lower bound will become Ω(log 1 ε ), closed to the O(log 2 1 ε )
upper bound shown in Theorem 4.
APPENDIX G PROOF OF COROLLARY 12
Proof. From Theorem 4, it follows that there exists a deep neural network f̃d of depth Ld = Θ ( log 1ε ) and size
Nd ≤ c ( log 1
ε
)2 (11)
for some constant c > 0 such that ∥f̃d − f∥ ≤ ε. From the equation (10) in the proof of Theorem 11, it follows that for all shallow neural networks f̃s of depth Ls and
∥∥∥f̃s − f∥∥∥ ≤ ε, their sizes should satisfy Ns ≥ Ls ( µ 16ε ) 1 2Ls ,
which is equivalent to
logNs ≥ logLs + 1
2Ls log ( µ 16ε ) . (12)
Substituting for log ( 1 ε ) from (12) to (11), we have
Nd = O(L2s log 2 Ns).
By definition, a shallow neural network has a small number of layers, i.e., Ls. Thus, the size of the deep neural network is O(log2 Ns). This means Nd ≪ Ns.
APPENDIX H PROOF OF COROLLARY 13
Corollary 13 (Gaussian function). For Gaussian function f(x) = f(x(1), ..., x(d)) = e− ∑d i=1(x (i))2/2, x ∈ [0, 1]d, there exists a deep neural network f̃(x) with O ( log dε ) lay-
ers, O ( d log dε ) binary step units and O ( d log dε + ( log 1ε )2) rectifier linear units such that
|f̃(x)− f(x)| ≤ ε for ∀x ∈ [0, 1]d.
Proof. It follows from the Theorem 4 that there exists d multilayer neural networks g̃1(x (1)), ..., g̃d(x (d)) with O ( log dε ) layers and O ( d log dε ) binary step units and O ( d log dε ) rectifier linear units in total such that∣∣∣∣∣x(1) 2 + ...+ x(d) 2 2 − g̃1(x (1)) + ...+ g̃d(x (d)) 2
∣∣∣∣∣ ≤ ε2 . (13) Besides, from Theorem 4, it follows that there exists a deep neural network f̂ with O ( log 1ε ) layers
O ( log 1ε ) binary step units and O (( log 1ε )2) such that
|e−dx − f̂(x)| ≤ ε 2 , ∀x ∈ [0, 1].
Let x = (g̃1(x(1)) + ...+ g̃d(x(d)))/2d, then we have∣∣∣∣∣e−(∑di=1 g̃i(x(i)))/2 − f̂ (∑d i=1 g̃i(x (i)) 2 )∣∣∣∣∣ ≤ ε2 . (14) Let the deep neural network
f̃(x) = f̂
( g̃1(x (1)) + ...+ g̃d(x (d))
2
) .
By inequalities (13) and (14), the the approximation error is upper bounded by
|f(x)− f̃(x)| = ∣∣∣∣∣e−(∑di=1 x(i))/2 − f̂ (∑d i=1 g̃i(x (i)) 2 )∣∣∣∣∣ ≤ ∣∣∣e−(∑di=1 x(i))/2 − e−(∑di=1 g̃i(x(i)))/2∣∣∣+ ∣∣∣∣∣e−(∑di=1 g̃i(x(i)))/2 − f̂ (∑d i=1 g̃i(x (i)) 2
)∣∣∣∣∣ ≤ ε
2 +
ε 2 = ε.
Now the deep neural network has O ( log dε ) layers, O ( d log dε ) binary step units and
O ( d log dε + ( log 1ε )2) rectifier linear units.
APPENDIX I PROOF OF COROLLARY 14
Corollary 14 (Ridge function). If f(x) = g(aTx) for some direction a ∈ Rd with ∥a∥1 = 1, a ⪰ 0, x ∈ [0, 1]d and some univariate function g satisfying conditions in Theorem 4, then there exists a multilayer neural network f̃ with O ( log 1ε ) layers, O ( log 1ε ) binary step units and O (( log 1ε )2) rectifier linear units such that |f(x)− f̃(x)| ≤ ε for ∀x ∈ [0, 1]d.
Proof. Let t = aTx. Since ∥a∥1 = 1, a ⪰ 0 and x ∈ [0, 1]d, then 0 ≤ t ≤ 1. Then from Theorem 4, it follows that then there exists a multilayer neural network g̃ with O ( log 1ε ) layers,
O ( log 1ε ) binary step units and O (( log 1ε )2) rectifier linear units such that
|g(t)− g̃(t)| ≤ ε, ∀t ∈ [0, 1].
If we define the deep network f̃ as f̃(x) = g̃(t),
then the approximation error of f̃ is
|f(x)− f̃(x)| = |g(t)− g̃(t)| ≤ ε.
Now we have proved the corollary.

SUMMARY
This paper contributes to the description and comparison of the representational power of deep vs shallow neural networks with ReLU and threshold units. The main contribution of the paper is to show that approximating a strongly convex differentiable function is possible with much less units when using a network with one more hidden layer.
PROS
The paper presents an interesting combination of tools and arrives at a nice result on the exponential superiority of depth.
CONS
The main result appears to address only strongly convex univariate functions.
SPECIFIC COMMENTS
- Thanks for the comments on L. Still it would be a good idea to clarify this point as far as possible in the main part. Also, I would suggest to advertise the main result more prominently.
I still have not read the revision and maybe you have already addressed some of these points there.
- The problem statement is close to that from [Montufar, Pascanu, Cho, Bengio NIPS 2014], which specifically arrives at exponential gaps between deep and shallow ReLU networks, albeit from a different angle. I would suggest to include that paper it in the overview.
- In Lemma 3, there is an i that should be x
- In Theorem 4, ``\tilde f'' is missing the (x).
- Theorem 11, the lower bound always increases with L ?
- In Theorem 11, \bf x\in [0,1]^d?