Low-shot visual learning, the ability to recognize novel object categories from very few, or even one example, is a hallmark of human visual intelligence. Though successful on many tasks, deep learning approaches tends to be notoriously datahungry. Recently, feature penalty regularization has been proved effective on capturing new concepts. In this work, we provide both empirical evidence and theoretical analysis on how and why these methods work. We also propose a better design of cost function with improved performance. Close scrutiny reveals the centering effect of feature representation, as well as the intrinsic connection with batch normalization. Extensive experiments on synthetic datasets, the one-shot learning benchmark “Omniglot”, and large-scale ImageNet validate our analysis.
1 INTRODUCTION
The current success of deep learning hinges on the ability to apply gradient-based optimization to high-capacity models. It has achieved impressive results on many large-scale supervised tasks such as image classification Krizhevsky et al. (2012); He et al. (2016) and speech recognition Yu & Deng (2012). Notably, these models are extensively hungry for data.
In contrast, human beings have strong ability to learn novel concepts efficiently from very few or even one example. As pointed out in Lake et al. (2016), human learning is distinguished by its richness and efficiency. To test whether machines can approach this goal, Lake et al. propose the invaluable “Omniglot” hand-written character classification benchmark Lake et al. (2011), where each training class has very few examples and the ability to fast-learn is evaluated on never-seen classes with only one example.
There has been previous work on attaining rapid learning from sparse data, denoted as meta-learning or learning-to-learn Thrun. (1998); Baxter. (1998). Although used in numerous senses, the term generally refers to exploiting meta-knowledge within a single learning system across tasks or algorithms. In theory, a meta-learning is able to identify the right “inductive bias shifts” from previous experiences given enough data and many tasks Baxter. (1998). However, even if a well-designed convolutional neural network is a good “inductive bias shift” for a visual recognition task, it is still elusive to find the optimal parameter from a small training set without any prior knowledge.
To alleviate this issue, low-shot learning methods have been proposed to transfer knowledge from various priors to avoid over-fitting, such as Bart & Ullman (2005). Recently, Hariharan & Girshick. (2016) propose a novel prior of gradient penalty, which works pretty well experimentally. Although an intuitive explanation is provided in Hariharan & Girshick. (2016) that a good solution of the network parameters should be stable with small gradients, it is mysterious why adding such a regularization magically improves the low-shot task by a large margin. Mathematical derivation shows that gradient penalty is closely related to regularizing the feature representation.
In this paper, we give more analysis, both empirically and theoretically, on why adding a gradient regularization, or feature penalty performs so well. Moreover, we carefully carry out two case studies: (1) the simplest non-linear-separable XOR classification, and (2) a two-layer linear network for regression. The study does give insight on how the penalty centers feature representations and
make the learning task easier. Furthermore, we also theoretically show that adding another finallayer weight penalty is necessary to achieve better performance. To be added, close scrutiny reveals its inherent connection with batch normalization Ioffe & Szegedy (2015). From a Bayesian point of view, feature penalty essentially introduces a Gaussian prior which softly normalizes the feature representation and eases the following learning task.
1.1 RELATED WORK
There is a huge body of literature on one-shot learning and it is beyond the scope of this paper to review the entire literature. We only discuss papers that introduce prior knowledge to adjust the neural network learning process, as that is the main focus of this work.
Prior knowledge, or “inductive bias” Thrun. (1998), plays an important role in one-shot or low-shot learning. To reduce over-fitting problems, we need to regularize the learning process. Common techniques include weight regularization Bishop (1995). Recently in Hariharan & Girshick. (2016), a gradient penalty is introduced, which works well experimentally in low-shot scenarios.
Various forms of feature regularization have been proposed to improve generalization: Dropout Srivastava et al. (2014) is effective to reduce over-fitting, but has been eschewed by recent architectures such as batch-normalization Ioffe & Szegedy (2015) and ResNets He et al. (2016). Other forms have also been proposed to improve transfer learning performance, such as minimizing the correlation of features Cogswell et al. (2016) and the multiverse loss Littwin & Wolf (2015) .
Our work is also closely related to metric learning and nearest neighbor methods, in which representations from previous experience are applied in cross-domain settings Fink (2005); Koch et al. (2015); Goldberger et al. (2005); Chopra et al. (2005). The insight lies in that a well-trained representational model have strong ability to generalize well on new tasks. In a recent work Santoro et al. (2016), DeepMind proposed a Memory Augmented Neural Network (MANN) to leverage the Neural-Turing-Machine for one-shot tasks. However, in Vinyals et al. (2016), it is found that a good initialization such as VGG-Net largely improves the one-shot performance. In our opinion, a good feature representations still play a central role in low-shot tasks.
1.2 CONTRIBUTIONS
The main contributions of our comprehensive analysis are three-fold:
1. We carefully carry out two case studies on the influence of feature regularization on shallow neural networks. We observe how the regularization centers features and eases the learning problem. Moreover, we propose a better design to avoid degenerate solutions.
2. From Bayesian point of view, close scrutiny reveals internal connections between feature regularization and batch normalization.
3. Extensive experiments on synthetic, the “Omniglot” one-shot and the large-scale ImageNet datasets validate our analysis.
2 AN ANALYSIS OF FEATURE REGULARIZATION
We briefly introduce the notations in our work: we denote uppercase A, bold a and lowercase a for matrices, vectors and scalars respectively. For a vector ai, we denotes ai,j as its j-th element. ||.||F stands for the Frobenius norm of a matrix. Given N examples {(xi, yi)|i = 1, ..., N}, we define E{.} as an expectation taken with respect to the empirical distribution generated by the training set. Following Hariharan & Girshick. (2016), we aim to learn a neural network model to extract the feature representations φ(xi) and make predictions ŷi = Wφ(xi) with W = [w1, ...,w|C|]. This setting includes both classification and regression problems, with |C| as the number of classes or target dimension, respectively. The problem can be generally formulated as:
W ∗, φ∗ = arg min W,φ E{l(W,φ(xi), yi)} (1)
where l(.) can be any reasonable cost function. In this paper, we focus on cross-entropy and L2 loss due to their convexity and universality.
In Hariharan & Girshick. (2016), it is suggested that adding a squared gradient magnitude loss (SGM) on every sample can regularize the learning process.
W ∗, φ∗ = arg min W,φ E{l(W,φ(xi), yi) + λ||∇W l(W,φ(xi, yi))||2} (2)
The insight is that for a good solution, the parameter gradient should be small at convergence. However, we know that the convergence of a neural network optimization is a dynamic equilibrium. In other words, at a stationary point, we should have E{∇W l(W,φ(x))} → 0. Intuitively when close to convergence, about half of the data-cases recommend to update a parameter to move positive, while the other half recommend to move negative. It is not very clear why small gradients on every sample E{||∇W l(W,φ(x))||2} produces good generalization experimentally. Mathematical derivation shows that the optimization problem with gradient penalty is equivalent with adding a weighted L2 regularizer φ(xi):
arg min W,φ
E{l(W,φ(xi), yi) + λαi||φ(xi)||2} (3)
where the example-dependent αi measures the deviation between the prediction ŷi and the target yi. In a regression problem, we have αi = r2 = ||ŷi − yi||2, with the residual r = ŷi − yi; in a classification problem, we have αi = ∑ k(p i k − I(yi = k))2. Intuitively, the misclassified high-norm examples might be outliers, and in a low-shot learning scenario, such outliers can pull the learned weight vectors far away from the right solution. In Hariharan & Girshick. (2016), the authors compare dropping αi and directly penalizing ||φ(xi)||2, which performs almost equally well. In our work, we argue that a better and more reasonable design should be:
arg min W,φ
E{l(W,φ(xi), yi) + λ1αi||φ(xi)||2}+ λ2||W ||2F (4)
where we add another weight regularizer ||W ||2F , which is necessary to avoid degenerate solutions. We will give further explanation in our second case study. In following analysis, we denote the cost in Eqn(4) with example-dependent αi as weighted L2 feature penalty, and the example-independent (setting αi ≡ 1) as uniform L2 feature penalty. We carry out two case studies: (1) an XOR classification and (2) a regression problem, both empirically and theoretically to analyze how the uniform and weighted L2 feature penalty regularize the neural network. In our paper, we will focus on the uniform feature regularization and will also cover the weighted scenario as well.
2.1 CASE STUDY 1: AN EMPIRICAL ANALYSIS OF XOR CLASSIFICATION
First, we study the simplest linear-non-separable problem– exclusive-or (XOR). Suppose that we have four two-dimensional input points x = [x1, x2]T ∈ R2: {x1 = [1, 1]T ,x2 = [0, 0]T } ∈ C+ belongs to the positive class, while {x3 = [1, 0]T ,x4 = [0, 1]T } ∈ C− the negative. As shown in Figure 1, we use a three-layer neural network to address the problem (left figure): h1 = x + b is a translation with b = [b1, b2]T ∈ R2 as the offset; h2 = h1,1 ∗ h1,2 is a non-linear layer, multiplying the first and second dimension of h1 and producing a scalar; y is a linear classifier on h2 parametrized by w. The original classification problem can be formulated as:
arg min w1,b,w 4∑ i=1 log(1 + exp(−yi ∗ w ∗ hi2))
Suppose that we start from an initialization b = [0, 0]T , all three samples {x2,x3,x4} from different classes will produce the same representation h2 = 0, which is not separable at all. It takes efforts to tune the learning rate to back-propagate w,b updates from the target y.
However, if we introduce the uniform L2 feature regularization as:
arg min w1,b,w 4∑ i=1 log(1 + exp(−yiwhi2)) + λ1 2 ||hi2||2
Then, we have:
λ1 2
∂||h2||2
∂b = λ1 ( E{h2(x2 + b2)} E{h1(x1 + b1)} ) (5)
the gradient descent pulls Eqn(5) towards zero, i.e., pulling b towards b1 = −E{x1} = −0.5 and b2 = −E{x2} = −0.5. As shown on the right of Figure 1, the gradient of feature regularization pulls h1 along the direction of red arrows. Then, we have h2 > 0 for positive examples and h2 < 0 for negative ones, which means h2 is linearly-separable. In summary, we can observe that: Empirically, the feature regularization centers the representation h2 = φ(x) and makes the following classification more learnable. For the weighted case, the offset b have similar effects. It can be derived that when converged the feature representation will satisfy E{h1} = 0 and E{h2}=0.
2.2 CASE STUDY 2: A COMPREHENSIVE ANALYSIS ON A REGRESSION PROBLEM
Next, we analyze a two-layer linear neural network as shown in Figure 2. Denoting the input as X = [x1,x2, ...] and the target as Y = [y1,y2, ...]. The regression loss can be formulated as:
E{||y −W2W1x||2} where the latent feature is h = φ(x) = W1x. The optimization of {W1,W2} in this multi-layer linear neural network is not trivial, since it satisfies following properties:
1. The regression loss is non-convex and non-concave. It is convex on W1 (or W2) when the other parameter W2 (or W1) is fixed, but not convex on both simultaneously;
2. Every local minimum is a global minimum; 3. Every critical point that is not a global minimum is a saddle point;
4. IfW2∗W1 is full-rank, the Hessian at any saddle point has at least one negative eigenvalue.
We refer interested readers to Baldi & Hornik (1989); Kawaguchi (2016) for detailed analysis.
In case of the uniform L2 feature penalty, the problem becomes:
E(W1,W2) = E{ 1
2 ||y −W2W1x||2 + λ1 2 ||W1x||2}+ λ2 2 ||W2||2F (6)
At the global minimum {W ∗1 ,W ∗2 }, we should have: ∂E
∂W1 |W∗1 = W T 2 Σxy −WT2 W2W1Σxx + λ1W1Σxx = 0 (7)
∂E
∂W2 |W∗2 = ΣxyW T 1 −W2W1ΣxxWT1 + λ2W2 = 0 (8)
where we define the variance and covariance matrix as Σxx = E{xxT }, Σxy = E{yxT }. Carrying out Eqn(7) ∗WT1 −WT2 ∗ Eqn(8) = 0 reveals a very interesting conclusion:
λ1E{||W1x||2} = λ2||W2||2F (9) This reads as the expected L2 feature penalty should be equal to final-layer weight regularizer when converged. Or equivalently, when close to convergence, the L2 feature penalty reduces overfitting by implicitly penalizing the corresponding weight matrix W . A more generalized form is:
Lemma 1 For a cost function of form in Eqn (4) with uniform L2 feature regularization:
arg min W,φ
E{l(W,φ(xi), yi) + λ1||φ(xi))||2}+ λ2||W ||2F
we have: λ1E{||φ(x)||2} = λ2||W ||2F (10)
The φ(.) can take a quite general form of a convolutional neural network with many common nonlinear operations such as the ReLU, max-pooling and so on. One can follow the derivation of Eqn(9) to easily derive Lemma 1.
Lemma 1 also reveals the importance of adding the weight penalty ||W ||2F in Eqn(4). If we only include the the feature penalty and drop the weight penalty (λ2 = 0 in our case), then a scaling as φ(.) = γφ(.) and W = 1γW with γ < 1 will always decrease the energy and the solution will become very ill-conditioned with γ → 0.
2.2.1 L2 FEATURE REGULARIZATION MAKES OPTIMIZATION EASIER
Moreover, we analyze numerically how the L2 feature penalty influences the optimization process in our regression problem.
We study a special case {x ∈ Rd, y ∈ R}withW1 ∈ R1×m,W2 ∈ R and include offsets b1 ∈ R and b2 ∈ R to make the problem more general. Then, the latent representation becomes h = W1x + b1 and the prediction is ŷ = W2h+ b2. The cost function of Eqn(4) becomes:
E(W1, b1,W2, b2) = 1
2 E{(W2h+ b2 − y)2 + λ1 2 r2†h 2}+ λ2 2 W 22 (11)
We define the prediction residual r and r† as ŷ − y for better readability, and substitute αi = (ri†)2. Numerically, we apply a two-step process: in the first step, we calculate the sample-dependent ri† in the feed-forward process to obtain our L2 feature regularization weights αi for each (xi, yi); in the second step, we treat r† as a constant in the optimize. The gradient and Hessian matrix of Eqn(11) can be derived as: ∂E
∂W2 = E{rhT }+ λ2W2
∂E ∂b2 = E{r} ∂E ∂hi = W2r i + λ1(r i †) 2hi
∂E
∂W1 = W2E{rxT }+ λ1E{r2†hxT }
∂E ∂b1 = W2E{r}+ λ1E{r2†h}
and ∂2E
∂W 22 = E{hhT }+ λ2
∂2E ∂b22 = 1
∂2E
∂(hi)2 = W 22 + λ1(r i †) 2
∂2E ∂W 21 = (W 22 + λ1E{r2†})E{xxT } ∂2E ∂b21 = W 22 + λ1E{r2†}
Suppose that we apply a second-order optimization algorithm for the network parameter θ ∈ {w1, w2, b1, b2}, the updates should be M θ = −η ∗ (∂2E/∂θ2)−1(∂E/∂θ) with η > 0 as the step-size. If we unluckily start from a bad initialization point W2 → 0, the updates of hi, W1 and b1 are of the form:
M hi = −η(W 22 + λ1(ri†)2})−1(W2ri + λ1(ri†)2hi) M w1 = −η[(W 22 + λ1E{r2†})E{xxT }]−1
(W2E{r}+ λ1E{r2†h})E{x} M b1 = −η(W 22 + λ1E{r2†})−1(W2E{r}+ λ1E{r2†h})
The updates will become very ill-conditioned without the regularization term (λ1 = 0), since spectrum of the Hessian matrix is two-orders of infinitesimal O(W 22 ) and the gradient is of one-order O(W2). In comparison, with a reasonable choice of λ1 > 0, the computation can be largely stabilized when E{r2†} 6= 0.
When the algorithm finally converges to a local minimum {W ∗1 , b∗1,W ∗2 , b∗2}, the expectation of parameter and latent feature should have gradient close to 0:
∂E
∂W2 = E{rhT }+ λ2W2 → 0
∂E ∂b2 = E{r} → 0
Substituting this in the analysis of b1, we have: ∂E
∂b1 = W2E{r}+ λ1E{αh} =⇒ E{αh} → 0 (12)
In other words, the feature penalty centralizes the final hidden layer representation h(x) = φ(x). Especially, in the uniformL2-feature penalty case, we simply drop α in Eqn (12) and have E{h} = E{φ(x)} → 0. In summary, the feature penalty improves the numerical stability of the optimization process in the regression problem. The conclusion also holds for the classification.
Similar results for φ(x) can be extended to deeper multilayer perceptrons with convex differentiable non-linear activation functions such as ReLU and max-pooling. In an m-layer model parametrized by {W1,W2, ...,Wm}, the Hessian matrix of hidden layers becomes strongly convex by backpropagating from the regularizer ||σm−1(Wm−1 ∗ σm−2(W2 ∗ (... ∗ σ1(W1x))))||2.
2.3 UNIFORM L2 FEATURE PENALTY IS A SOFT BATCH NORMALIZATION
In our two case studies, we can observe that L2 feature penalty centers the representation φ(x), which reminds us of the batch normalization Ioffe & Szegedy (2015) with similar whitening effects. We reveal here that the two methods are indeed closely related in spirit.
From the Bayesian point view, we analyze a binary classification problem: the probability of prediction ŷ given x observes a Bernoulli distribution p(ŷ|x, φ,W ) = Ber(ŷ|sigm(Wφ(x)), where φ(x) ∈ Rd is a neural network parametrized by φ and W ∈ R1×d. Assuming a factorized Gaussian prior on φ(x) ∼ N(0,Diag(σ21)) and W ∼ N(0,Diag(σ22)), we have the posterior as:
p(φ,W |{(xi, yi)}) = p({(xi, yi)}|φ,W )p(φ)p(W )
∝ ∏ i ( 1 1 + e−Wφ(xi) )y i ( 1 1 + eWφ(xi) )1−y i exp(− ||φ(x i)||2 2σ21 ) ( √ 2πσ1)d exp(− ||W || 2 F 2σ22 ) ( √ 2πσ2)d
(13)
Applying the maximum-a-posteriori (MAP) principle to Eqn(13) leads to the following objective:
E{l(W,φ(xi), yi) + 1 2σ21 ||φ(xi)||2}+ 1 2σ22 ||W ||2F + C (14)
with C = −d ln( √ 2πσ1) − d ln( √
2πσ2) is a constant. This is exactly the uniform L2 version of Equation (4) with λ1 = 12σ21 , λ2 = 1 2σ22
and αi = 1. The i.i.d. Gaussian prior on W and φ(x) has the effect of whitening the final representation.
The difference between uniform L2 feature penalty and batch normalization is that the former whitens φ(x) implicitly with an i.i.d. Gaussian prior during training, while the latter explicitly normalizes the output by keeping moving average and variance. We could say that the uniform L2 penalty is a soft batch normalization.
As analyzed in Wiesler & Ney (2011) this kind of whitening improved the numerical behavior and makes the optimization converge faster. In summary, the L2 feature regularization on φ(x) indeed tends to reduce internal covariate shift.
2.4 FEATURE REGULARIZATION MAY IMPROVE GENERALIZATION
As pointed out in Hariharan & Girshick. (2016), the gradient penalty or feature regularization improves the performance of low-shot learning experimentally. Here, we give some preliminary analysis on how and why our modified model in Eqn 4 may improve generalization performance in neural network learning. Denoting the risk functional (testing error) R(W ) as:
R(W ) = 1 2 E{||y −Wφ(x)||2} = 1 2 ∫ (x,y)∼P (x,y) ||y −Wφ(x)||2dP (x, y)
and empirical risk function (training error) Remp(W ) on {(xi, yi)|i = 1, ..., N} as:
Remp(W ) = 1
2N N∑ i=1 ||yi −Wφ(xi)||2
As we know, the training and testing discrepancy depends on the model complexity Vapnik (1998):
R(W )−Remp(W ) ≤ ν(W ) +O∗( h− ln η N ) (15)
where O∗(.) denotes order of magnitude up to logarithmic factor. The upper bound 15 holds true with probability 1 − η for a chosen 0 < η < 1. In the equation, h is a non-negative integer named the VC-dimension. The right side of Eqn 15 contains two term: the first one ν(W ) is the error on training set while the second one models complexity of the learning system. In a multilayer neural network with ρ parameters and non-linear activations, the system has a VC dimension Sontag (1998) of O(ρ log ρ).
In our model in Eqn 4, the final cost function includes both feature and weight penalty as:
arg min W,φ
E{l(W,φ(xi), yi) + λ1αi||φ(xi)||2}+ λ2||W ||2F
Empirically, the term λ2||W ||2F enforces large margin which limits the selection space of W . The term λ1||φ(xi)||2 not only improves numerical stability by feature whitening as discussed in the above sections, but also limits the selection of hidden layer parameter. These regularization terms thus reduces the VC-dimension of our learning. According to Eqn 15, the reduction of VC dimension of our model further reduces the theoretical discrepancy of training and testing performance.
3 EXPERIMENTAL RESULTS
We evaluate our algorithm on three datasets: synthetic XOR datasets, the Omniglot low-shot benchmark and the large-scale ImageNet dataset. We use our own implementation of SGM approach Hariharan & Girshick. (2016) for a fair comparison.
3.1 SYNTHETIC DATASETS
We first evaluate our model on the XOR dataset. Without loss of generality, we assume the data points x = [x1, x2]T are uniformly sampled from a rectangle x1 ∈ [−1, 1], x2 ∈ [−1, 1], and y(x) = I(x1 ∗ x2 > 0). The structure we use is a two-layer non-linear neural network with one
latent layer h = σ(W1x+b) where σ(.) is the rectified linear unit. ADAM Kingma & Ba (2014) is leveraged to numerically optimize the cross entropy loss.
During training, we make use of only 4 points {[1, 1], [−1, 1], [1,−1], [−1,−1]}, while randomly sampled points from the whole rectangle as the test set. It is a very low-shot task. As shown in Figure 3, our model with both feature and weight regularization outperforms the gradient penalty Hariharan & Girshick. (2016) and no regularization. We use uniform feature penalty as in Eqn(4) and set λ1 = λ2 = 0.1 in our experiments.
3.2 LOW-SHOT LEARNING ON OMNIGLOT
Our second experiment is carried out on the Omniglot one-shot benchmark Lake et al. (2011). Omniglot training set contains 964 characters from different alphabets with only 20 examples per each character. The one-shot evaluation is a pairwise matching task on completely unseen alphabets.
Following Vinyals et al. (2016), we use a simple yet powerful CNN as feature representation model, consisting of a stack of modules, each of which is a 3 × 3 convolution with 128 filters followed by batch normalizationIoffe & Szegedy (2015), a ReLU and 2× 2 max-pooling. We resized all images to 28×28 so that the resulting feature shape satisfies φ(x) ∈ R128. A fully connected layer followed by a softmax non-linearity is used to define the Baseline Classifier.
We set λ1=1e-4 in SGM Hariharan & Girshick. (2016) and λ1=λ2=1e-4 in our model. A nearest neighbor approach with L2 distance of feature φ(x) is applied for one-shot evaluation. As shown in Table 1, we can see that our model with both feature and weight penalty is able to achieve satisfactory performance of one-shot 91.5% accuracy, highly competitive with the state-of-the-art Matching NetworkVinyals et al. (2016) with CNN warm-start and RNN hyper-parameter tuning.
3.3 LARGE-SCALE LOW-SHOT LEARNING ON IMAGENET
Our last experiment is on the ImageNet benchmark Russakovsky et al. (2015). It contains a wide array of classes with significant intra-class variation. We divide the 1000 categories randomly into 400 base for training and evaluate our feature representation on the 600 novel categories.
We use a 50-layer residual network He et al. (2016) as our baseline. Evaluation is measured by top-1 accuracy on the 600 test-set in a 20-way setup, i.e., we randomly choose 1 sample from 20 test classes, and applies a nearest neighbor matching. As shown in Table 2, we can see that our model learns meaningful representations for unseen novel categories even with large intra-class variance.
3.4 COMPARISON WITH BATCH-NORMALIZATION
As discussed in Section 2.3, feature penalty has similar effects with batch normalization Ioffe & Szegedy (2015). It is of interest to compare the influence of two modules influence training performance of neural networks. We study the performance of the classification with and without each modules on four classic image classification benchmarks. For CIFAR-10 and ImageNet, we applied the Residual Net architecture He et al. (2016), while stacked convolution layers with ReLU and max-pooling is applied for MNIST and Omniglot. For ImageNet benchmark evaluation, we test the top-1 accuracy on the validation set with 50,000 images.
Since in our model the feature penalty regularizer is applied only on the last hidden layer, we still keep the batch-normalization modules in previous layers in our “FP” model. As shown in Figure 4, we observe that baseline models with neither “BN” nor “FP” takes much longer to converge and achieve inferior performance; our “FP” regularizer achieves almost the same performance on MNIST (both 99%), CIFAR-10 (both 93%) and Omniglot 1-shot (88% BN v.s. 89% FP); on ImageNet, “BN” performs better than our “FP” (75% BN v.s. 74% FP). With both batch-normalization
and feature penalty modules added, we achieve the best classification performance on all four benchmarks.
4 CONCLUSION
In this work, we conduct an analysis, both empirically and theoretically, on how feature regularization influences and improves low-shot learning performance. By exploiting an XOR classification and two-layer linear regression, we find that the regularization of feature centers the representation, which in turn makes the learning problem easier with better numerical behavior. From the Bayesian point of view, the feature regularization is closely related to batch normalization. Evaluation on synthetic, “Omniglot” one-shot and large-scale ImageNet benchmark validates our analysis.

Summary
===
This paper extends and analyzes the gradient regularizer of Hariharan and
Girshick 2016. In that paper a regularizer was proposed which penalizes
gradient magnitudes and it was shown to aid low-shot learning performance.
This work shows that the previous regularizer is equivalent to a direct penalty
on the magnitude of feature values weighted differently per example.
The analysis goes to to provide two examples where a feature penalty
favors a better representation. The first example addresses the XOR
problem, constructing a network where a feature penalty encourages
a representation where XOR is linearly separable.
The second example analyzes a 2 layer linear network, showing improved stability
of a 2nd order optimizer when the feature penalty is added.
One last bit of analysis shows how this regularizer can be interpreted as
a Gaussian prior on both features and weights. Since the prior can be
interpreted as having a soft whitening effect, the feature regularizer
is like a soft version of Batch Normalization.
Experiments show small improvements on a synthetic XOR test set.
On the Omniglot dataset feature regularization is better than most baselines,
but is worse than Moment Matching Networks. An experiment on ImageNet similar
to Hariharan and Girshick 2016 also shows effective low-shot learning.
Strengths
===
* The core proposal is a simple modification of Hariharan and Girshick 2016.
* The idea of feature regularization is analyzed from multiple angles
both theoretically and empirically.
* The connection with Batch Normalization could have broader impact.
Weaknesses
===
* In section 2 the gradient regularizer of Hariharan and Girshick is introduced.
While introducing the concept, some concern is expressed about the motivation:
"And it is not very clear why small gradients on every sample produces
good generalization experimentally." This seems to be the central issue to me.
The paper details some related analysis, it does not offer a clear answer to
this problem.
* The purpose and generality of section 2.1 is not clear.
The analysis provides a specific case (XOR with a non-standard architecture)
where feature regularization intuitively helps learn a better representation.
However, the intended take-away is not clear.
The take-away may be that since a feature penalty helps in this case it
should help in other cases. I am hesitant to buy that argument because of the
specific architecture used in this section. The result seems to rely on the
choice of an x^2 non-linearity, which is not often encountered in recent neural
net literature.
The point might also be to highlight the difference between a weight
penalty and a feature penalty because the two seem to encourage
different values of b in this case. However, there is no comparison to
a weight penalty on b in section 2.1.
* As far as I can tell, eq. 3 depends on either assuming an L2 or cross-entropy
loss. A more general class of losses for which eq. 3 holds is not provided. This
should be made clear before eq. 3 is presented.
* The Omniglot and ImageNet experiments are performed with Batch Normalization,
yet the paper points out that feature regularization may be similar in effect
to Batch Norm. Since the ResNet CNN baseline includes Batch Norm and there are
clear improvements over that baseline, the proposed regularizer has a clear
additional positive effect. However, results should be provided without
Batch Norm so a 1-1 comparison between the two methods can be performed.
* The ImageNet experiment should be more like Hariharan and Girshick.
In particular, the same split of classes should be used (provided in
the appendix) and performance should be measured using n > 1 novel examples
per class (using k nearest neighbors).
Minor:
* A brief comparison to Matching Networks is provided in section 3.2, but the
performance of Matching Networks should also be reported in Table 1.
* From the approach section: "Intuitively when close to convergence, about half
of the data-cases recommend to update a parameter to go left, while
the other half recommend to go right."
Could the intuition be clarified? There are many directions in high
dimensional space and many ways to divide them into two groups.
* Is the SGM penalty of Hariharan and Girshick implemented for this paper
or using their code? Either is acceptable, but clarification would be appreciated.
* Should the first equal sign in eq. 13 be proportional to, not equal to?
* The work is dense in nature, but I think the presentation could be improved.
In particular, more detailed derivations could be provided in an appendix
and some details could be removed from the main version in order to increase
focus on the results (e.g., the derviation in section 2.2.1).
Overall Evaluation
===
This paper provides an interesting set of analyses, but their value is not clear.
There is no clear reason why a gradient or feature regularizer should improve
low-shot learning performance. Despite that, experiments support that conclusion,
the analysis is interesting by itself, and the analysis may help lead to a
clearer explanation.
The work is a somewhat novel extension and analysis of Hariharan and Girshick 2016.
Some points are not completely clear, as mentioned above.


ON ROBUST CONCEPTS AND SMALL NEURAL NETS
The universal approximation theorem for neural networks says that any reasonable function is well-approximated by a two-layer neural network with sigmoid gates but it does not provide good bounds on the number of hidden-layer nodes or the weights. However, robust concepts often have small neural networks in practice. We show an efficient analog of the universal approximation theorem on the boolean hypercube in this context. We prove that any noise-stable boolean function on n boolean-valued input variables can be well-approximated by a two-layer linear threshold circuit with a small number of hidden-layer nodes and small weights, that depend only on the noisestability and approximation parameters, and are independent of n. We also give a polynomial time learning algorithm that outputs a small two-layer linear threshold circuit that approximates such a given function. We also show weaker generalizations of this to noise-stable polynomial threshold functions and noise-stable boolean functions in general.
1 INTRODUCTION
The universal approximation theorem of Hornik et al. (1989) and Cybenko (1992) provides a foundation to the mathematical theory of artificial neural networks. It states that any continuous function on a compact subset of the Euclidean space can be approximated arbitrarily well by a feed-forward artificial neural network with only one hidden layer containing finitely many neurons, under mild assumptions on the activation function. In such neural networks, each node applies an activation function to a weighted linear combination of its inputs, and the above theorem holds true for many different choices of activation functions as shown by Hornik (1991). However, the universal approximation theorem and its quantitative improvements by Barron (1993) and others have certain limitations, namely, they do not provide reasonable, practical bounds or efficient learning algorithms for the parameters of these neural networks, that is, the number of neurons in the hidden layer and the size of weights used in the linear combinations. For a detailed survey of these results in approximation theory, we point the reader to Pinkus (1999).
In practice, we notice that even moderate-sized neural networks can be trained to learn various natural concepts in computer vision tasks, and the typical rules of thumb followed for their model and size selection are usually guided by the domain knowledge, the learning algorithm, and the available computational resources more than any theoretical bounds; see Simard et al. (2003). The known theoretical bounds are either based on the Network Information Criterion (NIC) by Amari (1998), which is a generalization of Akaike Information Criterion (AIC) by Akaike (1974) used in statistical inference, or based on the Vapnik-Chervonenkis dimension; see Baum & Haussler (1989), Bartlett (1993), Maass (1995), Karpinski & Macintyre (1997). These bounds do not adequately explain the observed efficiency of learning many natural concepts in practice.
∗This work was done during an internship at Microsoft Research India, when the author was a student at Chennai Mathematical Institute, H1, SIPCOT IT Park, Siruseri, Chennai 603103, India
Most natural concepts are often based on a small number of relevant attributes or features, and can be learnt efficiently once we implicitly map our input to the correct attribute space and focus on these relevant attributes or features. Moreover, most natural concepts are also robust, that is, their positive and negative examples are reasonably unambiguous and far from each other. Thus, an important theoretical question is to understand the underlying cognitive process, find a reasonably close and accurate model for it, and answer why certain models like artificial neural networks can mimic this cognitive process in practice.
The implicit mapping of our input coordinates to the space of attributes is formalized by the kernel method in machine learning; see Hofmann et al. (2008). Attribute-efficient learning proposed by Valiant (2000) and Littlestone (1988) captures the ease of learning via improved VC-dimension bounds that depend only a small number of relevant attributes. Robust concepts are often defined using large-margin classifiers studied in the context of Support Vector Machines; see Cortes & Vapnik (1995). We use a different notion of robustness suited to the boolean hypercube known as noise-stability. Due to known results from Fourier analysis over the boolean hypercube, noisestability also implies closeness to a function that depends only on a small number of attributes.
Since the universal approximation theorem gives a depth-2 neural network with only one hidden layer, the effect of depth on the power of neural networks has attracted considerable interest in approximation theory as well as boolean circuit complexity; see de Villiers & Barnard (1993) and Siu et al. (1995). Note that on the boolean hypercube, depth-d circuits with sigmoid gates and linear threshold gates are essentially equivalent. An important result relevant to our paper is due to a long line of work including Goldmann et al. (1992), Goldmann & Karpinski (1998), and Hofmeister (1996) which proved that any depth-d linear threshold circuit with polynomially (in the number n of input variables) many nodes but arbitrary weights can be efficiently simulated by a depth-(d+ 1) linear threshold circuit with polynomially many nodes and polynomially bounded integer weights.
2 OUR RESULTS
We work with linear threshold circuits with boolean inputs and outputs, which are discrete analogs of the neural networks with real-valued inputs and continuous activation functions. They are also known as multi-layer perceptrons as in Minsky & Papert (1987), which are simply feed-forward neural networks where each node computes a weighted linear combination of its inputs and applies a threshold function for activation. As mentioned above, the notion of robustness we use is noisestability or low noise-sensitivity. The noise sensitivity of a boolean function is simply the fraction of inputs whose output changes, if we change each coordinate of the input independently with a small probability, say some > 0.
As a warm-up, we show that if a boolean function defined on the boolean hypercube {−1, 1}n is noise-stable, that is, if it has low noise-sensitivity, then it can be approximated by a depth-2 linear threshold circuit (that is, with one hidden layer), that depends only on constantly many variables in the input, and its number of hidden nodes and the weights are also constants, all independent of n. Here we quantify approximation or closeness based on the fraction of inputs where two functions differ. This result may be folklore although we are not aware of any reference.
Theorem 1. Any f : {−1, 1}n → {−1, 1} that has small noise-sensitivity for -perturbations, that is, NS (f) = O (δ √ ), is δ-close to a depth-2 linear threshold circuit that depends only on O(1) variables of the input with O(1) hidden nodes and O(1) weights, where the constants O(1) depend on and δ but are independent of n.
When the given function is actually a linear threshold function, that is, when it represents a halfspace, we can improve the above theorem with constants O(1) that are polynomial in 1/ and 1/δ, and thus, give an efficient analog of the universal approximation theorem for neural networks over the boolean hypercube. Note that this is consistent with the intuition that better noise-stable concepts can be approximated by smaller neural networks. It also shows that a given concept may be linearly separable in a high n-dimensional kernel space but its approximation by neural networks only depends on an inherent parameter like robustness or noise-sensitivity, independent of n.
Theorem 2. Any linear threshold function f : {−1, 1}n → {−1, 1} that has small noise-sensitivity for -perturbations, that is, NS (f) = O ( δ3 √ ) , is δ-close to a depth-2 linear threshold circuit
that depends only on O(1) variables of the input with O(1) hidden nodes and O(1) integer weights, where the constants are poly(1/ , 1/δ) but independent of n.
Equipped with this, we show the following implication for learning. Given oracle access to such a linear threshold function f of low noise-sensitivity, we can learn a depth-2 linear threshold circuit that approximates f well, in polynomial time. Theorem 3. Let f : {−1, 1}n → {−1, 1} be any linear threshold function with small noisesensitivity for -perturbations, that is, NS (f) = O ( δ3 √ ) . Then we can learn a depth-2 linear
threshold circuit on k variables that is exp ( −Ω( 3 √ log(1/δ)) ) -close to f with probability 1 − γ,
in time nk · p (1/ , 1/δ, 1/γ), where p is polynomial in 1/ , exponential in polylog (1/δ), and logarithmic in 1/γ, and k = O ( 1/ 2 · log(1/ ) · log(1/δ) ) . Moreover, the size and integer weights of
the depth-2 linear threshold circuits are polynomially bounded in 1/ and 1/δ.
We would also like to note that it is possible to extend our result for halfspaces to polynomial threshold functions. This uses the facts that any degree-d polynomial threshold function -close to a J-junta, is close to junta that is a polynomial threshold function of degree at most d, and that the machinery from De et al. (2014) extends to small weight polynomial threshold functions as well.
In a recent paper, Feldman & Vondrak (2013) have shown that sub-modular functions are close to O ( 1/ 2 · log (1/ ) ) -juntas. Note that this tells us that we can -approximate submodular func-
tions by polynomials of degree O ( 1/ 2 · log (1/ ) ) . This means we can approximate submodular functions by depth-3 neural networks with linear threshold gates everywhere except for the top gate.
2.1 OBSTACLES TO IMPROVEMENTS
We now discuss some obstacles to possible improvements of our results. The nk running time is needed to identify the specific set of O ( 1/ 2 · log(1/ ) · log(1/δ) ) relevant
coordinates. This nO(k) factor is unavoidable while learning k-juntas, and a candidate hard case is presented in Blum et al. (1994). Only recently Valiant (2015) gave an improved algorithm to learn k-juntas with noise rate η that runs in time less than n0.8k · poly ( 2k, 1/(1− 2η) ) .
Weak, proper, agnostic learning of halfspaces under non-uniform distributions is NP-hard as shown by Guruswami & Raghavendra (2006), and extended to improper learning by Daniely et al. (2013) and Daniely (2015). Daniely’s result rules out efficient, constant factor approximation for even improper learning of halfspaces using any hypothesis class on the boolean hypercube under nonuniform distributions1. However, Daniely (2014) can get around this by giving a PTAS for improper learning of halfspaces on the unit sphere under uniform distribution. Our result can be seen as another way to circumvent the hardness results. We learn noise-stable halfspaces on the boolean hypercube under uniform distribution, by giving an efficient, agnostic-type learning algorithm where the output hypothesis is a depth-2 neural network. This is arguably more natural than other improper learning results for halfspaces via low-degree polynomials.
Not having an efficient version of Bourgain’s theorem for arbitrary noise-stable boolean functions, where the number of junta variables is polynomial in the noise-sensitivity parameters is another obstacle to efficient generalizations of our result. Note that the proof of this for noise-stable halfspaces does not generalize to higher depth linear threshold circuits. Another approach is to approximate any noise-stable function first using a halfspace and then by a depth-2 linear threshold circuit, but this has been ruled out by Mossel & Neeman (2016) with an example of a noise-stable function that is far from any halfspace.
We now give a brief outline of the proofs of the above theorems. Bourgain (2002) proved that any function with small noise-sensitivity can be approximated by another function that is a junta, which means that it depends on very few coordinates. In Theorem 1, we show that such a function can also be represented by a small depth-2 linear threshold circuit with small size and small integer weights. Moreover, any linear threshold function that is close to a junta is actually close to a linear threshold
1Results in Daniely et al. (2013) are under certain assumptions that are refuted in Allen et al. (2015). However, Daniely (2015) recovers a slightly weaker but very similar result for halfspaces under different assumptions
function defined over those junta coordinates. Thus, we can approximate the given noise-stable function by a linear threshold function on a small number of inputs, however, its weights may be large. Therefore, we use the size-depth-weight trade-off from Goldmann et al. (1992) to simulate this linear threshold function by a depth-2 linear threshold circuit with small size as well as small weights in Theorem 2. We also use a recent improvement over Bourgain’s theorem by Diakonikolas et al. (2014) to get bounds polynomial in the noise-stability parameters. Theorem 3 follows by combining a result of De et al. (2014) on agnostic-type learning by a linear threshold function with a constructive, efficient simulation of the Goldmann et al. (1992) result by Goldmann & Karpinski (1998).
3 RELATED WORK
Motivated by the recent advances in neural networks, there have been various attempts to build a theory to understand why neural networks can efficiently simulate many natural concepts and why their models and parameters can be learnt efficiently, for example, Andoni et al. (2014) and Arora et al. (2014). Our objective is to show efficient analogs of the universal approximation theorem for neural networks, a question that has been studied in approximation theory as well as boolean circuit complexity. We combine the size-depth-weight trade-off results from about two decades ago such as Goldmann et al. (1992) and Goldmann & Karpinski (1998) with more recent work on the Fourier analysis of boolean functions and its corollaries in learning. Also note that There are known NP-hardness results for learning halfspaces by Guruswami & Raghavendra (2009) and for approximately learning depth-2 threshold circuits by Bartlett & Ben-David (2002). However, these are for arbitrary threshold circuits. As we will show, the noise-stability constraint allows us to get a polynomial time algorithm to learn a depth-2 threshold circuit approximating the original function.
The low effective-dimension of hyperparameters has been observed and exploited to learn using neural networks in practice by Bergstra & Bengio (2012). We propose noise-stability as an approach to study this theoretically.
Arriaga & Vempala (2006) showed that robust or large-margin halfspaces in Rn can be learnt efficiently using random projections. Their learning algorithm outputs a depth-2 neural network with different activation functions in different layers. We define robustness using noise-stability instead, and show that better noise-stability reduces learning complexity. Our results also generalize to polynomial threshold functions, that is, a noise-stable polynomial threshold function (PTF) can be represented by a small, depth-2 neural network.
4 PRELIMINARIES
Here we give a compilation of definitions and known results that we will use to prove Theorems 1, 2, and 3. Noise-stable boolean functions have low noise-sensitivity. Noise-sensitivity of a boolean function, with respect to -perturbations, is defined as the fraction of inputs whose output changes, when we change each bit of the input independently with a small probability .
Definition 1. The noise sensitivity of a boolean function f : {−1, 1}n → {−1, 1} at a given noise rate > 0 is defined as
NS (f) = Probx,y (f(x) 6= f(y)) ,
where x is uniformly distributed in {−1, 1}n, and y is obtained from x by flipping each bit of x independently with probability .
A theorem of Bourgain (2002) states that boolean functions with small noise-sensitivity are close to juntas, which are boolean functions that depend on very few coordinates. Note that the number of these relevant coordinates is independent of n. Lemma 1. Any f : {−1, 1}n → {−1, 1} that satisfies NS (f) = O (δ √ ) is δ-close to a k-junta, where
k =
( 1
δ
)O(1/ ) .
Here, δ-closeness means agreement on 1− δ fraction of the inputs. Note that the √ in the bound has a special significance for linear threshold functions, as we explain below. Definition 2. A linear threshold function f : {−1, 1}n → {−1, 1} is defined as
f(x) = sgn ( ∑n i=1 wixi − w0) ,
for some weights w0, w1, w2, . . . , wn ∈ R.
A theorem of Peres (2004) states that the noise sensitivity of any linear threshold function at noise rate is at most 2 √ . Lemma 2. Any linear threshold function f : {−1, 1}n → {−1, 1} satisfies NS (f) ≤ 2 √ .
The bounds in Proposition 1 can be improved when f is a linear threshold function as shown by the result of Diakonikolas et al. (2014) mentioned below. Thus, a noise-stable linear threshold function is close to a k-junta, where k is polynomial dependent on the noise and approximation parameters, but is independent of n. Lemma 3. Any linear threshold function f : {−1, 1}n → {−1, 1} that satisfies NS (f) = O ( δ(2− )/(1− ) √ ) , for some 0 < < 1/2, is δ-close to a k-junta, where
k = O
( 1
2 log
( 1 ) log ( 1
δ
)) .
Remark: For convenience, we use NS (f) = O ( δ3 √ )
in our assumption whenever using the above theorem.
The following lemma from O’Donnell & Servedio (2011) ties it up nicely to say that if any linear threshold function is close to a junta, then it must be close to a linear threshold function defined over those junta coordinates. Lemma 4. If a linear threshold function f : {−1, 1}n → {−1, 1} is δ-close to a junta over a subset J ⊆ [n] of coordinates, then f is δ-close to a linear threshold function defined over that subset J ⊆ [n] of coordinates.
Linear threshold circuits where each gate computes a linear threshold function forms an important class in circuit complexity. We borrow the standard definitions and notation from Siu et al. (1995) and Goldmann et al. (1992). Definition 3. LTd is defined as the class of linear threshold circuits of depth d on n inputs with the number of nodes polynomial in n but arbitrary weights inside the linear threshold functions. L̂T d is defined as the class of linear threshold circuit of depth d on n inputs with both the number of nodes and weights inside the linear threshold functions polynomially bounded in n.
The size-depth-weight trade-offs for linear threshold circuits have been studied in circuit complexity with keen interest, and a long line of work culminated in the following result by Goldmann et al. (1992). Here, the weight bounds are bounds on the ratio of the maximum and the minimum weights, when all of them are integers.
Lemma 5. LTd ⊆ L̂T d+1.
This means that any depth-d linear threshold circuit of polynomial size but arbitrary weights can be simulated by a depth-(d+ 1) linear threshold circuit whose size and weights are both polynomially bounded. While Goldmann et al. (1992) gives an existence result, Goldmann & Karpinski (1998) gives a constructive proof and it is easy to check that the underlying simulation is efficient and can be computed in polynomial time as well. Hofmeister (1996) has a simplified proof of Goldmann & Karpinski (1998) with improved explicit bounds.
Bourgain’s theorem has also been extended to the case of boolean functions with inputs that come from constant biased distributions over {−1, 1}n in Kindler & Safra (2002). Our general result can be extended to these cases as well. For this we need to define the λ-noise-sensitivity of a boolean function with respect to µp, where µp is the distribution that picks −1 with probability p and 1 with probability 1− p.
Definition 4. The λ-noise-sensitivity of a Boolean funciton f : {−1, 1}n → {−1, 1} with respect to µp is defined as
NSλ,p (f) = Probx,y (f(x) 6= f(y))
where x ∼ µnp and y is constructed by first sampling coordinates I from [n] according to µnλ and then replacing those coordinates in x by coordinates independently sampled from µIp.
Lemma 6. For any parameter λ > 0, fix k = log1−λ(1/2). Then every Boolean function f : {−1, 1}n → {−1, 1} whose λ-noise-sensitivity with respect to µnp is bounded by ( /k)2, is a max[O( log(1/p)/p2), J ]-junta, where
J = O
( k3
2pk
)
5 PROOF OF THEOREM 1
Proof. (Proof of Theorem 1) The proof immediately follows from Proposition 1 and the following easy lemma.
Lemma 7. Any f : {−1, 1}n → {−1, 1} that is a k-junta can be represented by a depth-2 linear threshold circuit with the number of nodes and weights bounded by 2O(k).
Proof. Since f is a k-junta we can pretend that f : {−1, 1}k → {−1, 1}. Each positive example x ∈ {−1, 1}k such that f(x) = 1 can be isolated by a single halfspace h(y) = sgn (〈x, y〉 − (k − 1/2)), which outputs positive value for y ∈ {−1, 1}k iff x = y. We can build a depth-2 linear threshold circuit where all the hidden nodes correspond to h(x), one for each positive examples of f . Thus, for a positive example of f , exactly one of the hidden layer node outputs 1. Otherwise, all hidden layer nodes output −1. Now we can have a linear threshold gate are the top with all weights 1 and threshold 1− p, where p is the number of positive examples of f . Note that all the hidden threshold gates have integer weights bounded by k and they are at most 2k in number. The top gate has integer weights bounded by 2k. Thus, f can be represented by an LT2 or depth-2 linear threshold circuit where the size of the circuit and the integer weights used in it are bounded by 2O(k).
Therefore, combining this with Proposition 1, we get that any noise-stable f as required in Theorem 1 is δ-close to a depth-2 linear threshold circuit whose size and integer weights are bounded by 2O(k), where
k =
( 1
δ
)O(1/ ) ,
independent of n.
6 PROOF OF THEOREM 2
Since Bourgain’s theorem can be improved for linear threshold functions with polynomial dependency in the noise and approximation parameters, we can approximate the given function using a junta where the number of junta variables is polynomially bounded. Due to Lemma 4, we can moreover, say that our function is not just close to a junta but close to a linear threshold function defined over these junta variables. The only caveat is that the weights used in this linear threshold function may be large. This is where we invoke size-depth-weight trade-off result such as Proposition 5 from circuit complexity to simulate this linear threshold function by a linear threshold circuit with an extra depth but polynomially bounded weights.
Proof. (Proof of Theorem 2) From Proposition 3, we see that any linear threshold function f with low noise-sensitivity NS (f) = O ( δ3 √ ) is δ-close to an O ( 1/ 2 log (1/ ) log (1/δ) ) -junta. From Lemma 4, moreover, it must be δ-close a linear threshold function over these junta variables.
Thus, f is δ-close to an LT1 function over these junta variables but the weights could be large. However, Proposition 5 shows that this can be simulated by an LT2 function over these junta variables with weights polynomially bounded in the number of junta variables. Therefore, f is δ-close to an LT2 function over O ( 1/ 2 log (1/ ) log (1/δ) ) variables with the size of the circuits and the weights at the threshold gates polynomially bounded in 1/ and 1/δ, but independent of n. This concludes the proof of Theorem 2.
7 PROOF OF THEOREM 3
Proof. (Proof of Theorem 3) Looking at Theorem 2, the broad outline of the algorithm is as follows. As seen in the proof of Theorem 2, we know that the given linear threshold function of low noisesensitivity is close to another linear threshold function that depends only on a small, constant number of input variables. We can go over each small subset by brute force. Now over each small subset, we can try to learn a linear threshold function over them that is closest to the given function. Here we use a result from De et al. (2014) (see Theorem 36 of De et al. (2014)) on agnostic-type learning halfspaces via reconstructing the Chow parameters of a linear threshold function; Chow parameters are the level-0 and level-1 Fourier coefficients which are known to completely determine a linear threshold function.
Lemma 8. Let f : {−1, 1}n → {−1, 1} and let opt be the minimum disagreement (in fraction of the inputs) of f with its closest linear threshold function. Then given any 0 < , γ < 1/2 and access to independent uniform samples (x, f(x)), we can output a linear threshold function g (given by its weights) such that, with probability 1− γ,
d(f, g) ≤ 2−Ω( 3 √ log(1/opt)) + ,
where the algorithm runs in time
Õ(n2) · ( 1 )O(log2(1/ )) · log ( 1
γ
) .
An immediate corollary that is useful to us is
Corollary 1. Let f : {−1, 1}n → {−1, 1} be a boolean function that is δ-close to a linear threshold function in a given subset S ⊆ [n] of k input variables. Then, for 0 < δ, γ < 1/2, and given access to independent uniform examples (x, f(x)), we can output a linear threshold function g (given by its weights) such that, with probability 1− γ,
d(f, g) ≤ 2−Ω( 3 √ log(1/δ)) + δ,
where the algorithm runs in time
Õ(k2)
( 1
δ
)O(log2(1/δ)) log ( 1
γ
) .
Thus, we go over all subsets of size O ( 1/ 2 · log(1/ ) · log(1/δ) ) and run the agnostic-type learning of linear threshold functions by De et al. (2014). We take the best of these and convert the corresponding output, which is a linear threshold function with weights possibly exponential in 1/ and 1/δ, and apply Goldmann & Karpinski (1998) to convert it into a depth-2 linear threshold circuit whose size and weights both are polynomially bounded in 1/ and 1/δ.
8 CONCLUSION AND FUTURE WORK
We show an efficient analog of the universal approximation theorem for neural networks in the case of noise-sensitive halfspaces of boolean hypercube, and gave efficient learning algorithms for the same. We do this via an interplay of techniques from Fourier analysis over the boolean hypercube and size-weight-depth trade-off results on linear threshold circuits from circuit complexity.
One might be able to extend these result to continuous domains where the input is sampled uniformly from [−1, 1]n by using the ANOVA (analysis of variance) decomposition of a function. However, to do this one will have to prove a Bourgain-type theorem for these settings.

SUMMARY
This paper presents a study of the number of hidden units and training examples needed to learn functions from a particular class.
This class is defined as those Boolean functions with an upper bound on the variability of the outputs.
PROS
The paper promotes interesting results from the theoretical computer science community to investigate the efficiency of representation of functions with limited variability in terms of shallow feedforward networks with linear threshold units.
CONS
The analysis is limited to shallow networks. The analysis is based on piecing together interesting results, however without contributing significant innovations.
The presentation of the main results and conclusions is somewhat obscure, as the therein appearing terms/constants do not express a clear relation between increased robustness and decreasing number of required hidden units.
COMMENTS
- In the abstract one reads ``The universal approximation theorem for neural networks says that any reasonable function is well-approximated by a two-layer neural network with sigmoid gates but it does not provide good bounds on the number of hidden-layer nodes or the weights.''
In page 1 the paper points the reader to a review article. It could be a good idea to include also more recent references.
Given the motivation presented in the abstract of the paper it would be a good idea to also comment of works discussing the classes of Boolean functions representable by linear threshold networks.
For instance the paper [Hyperplane Arrangements Separating Arbitrary Vertex Classes in n-Cubes. Wenzel, Ay, Paseman] discusses various classes of functions that can be represented by shallow linear threshold networks and provides upper and lower bounds on the number of hidden units needed for representing various types of Boolean functions. In particular that paper also provides lower bounds on the number of hidden units needed to define a universal approximator.
- It certainly would be a good idea to discuss the results on the learning complexity in terms of measures such as the VC-dimension.
- Thank you for the explanations regarding the constants.
So if the noise sensitivity is kept constant, larger values of epsilon are associated with a smaller value of delta and of 1/epsilon.
Nonetheless, the description in Theorem 2 is in terms of poly(1/epsilon, 1/delta), which still could increase?
Also, in Lemma 1 reducing the sensitivity at a constant noise increases the bound on k?
- The fact that the descriptions are independent of n seems to be related to the definition of the noise sensitivity as an expectation over all inputs. This certainly deserves more discussion. One good start could be to discuss examples of functions with an upper bound on the noise sensitivity (aside from the linear threshold functions discussed in Lemma 2).
Also, reverse statements to Lemma 1 would be interesting, describing the noise sensitivity of juntas specifically, even if only as simple examples.
- On page 3 ``...variables is polynomial in the noise-sensitivity parameters'' should be inverse of?
MINOR COMMENTS
On page 5 Proposition 1 should be Lemma 1?


RENDERGAN: GENERATING REALISTIC LABELED DATA
Deep Convolutional Neuronal Networks (DCNNs) are showing remarkable performance on many computer vision tasks. Due to their large parameter space, they require many labeled samples when trained in a supervised setting. The costs of annotating data manually can render the use of DCNNs infeasible. We present a novel framework called RenderGAN that can generate large amounts of realistic, labeled images by combining a 3D model and the Generative Adversarial Network framework. In our approach, image augmentations (e.g. lighting, background, and detail) are learned from unlabeled data such that the generated images are strikingly realistic while preserving the labels known from the 3D model. We apply the RenderGAN framework to generate images of barcode-like markers that are attached to honeybees. Training a DCNN on data generated by the RenderGAN yields considerably better performance than training it on various baselines.
Deep Convolutional Neuronal Networks (DCNNs) are showing remarkable performance on many computer vision tasks. Due to their large parameter space, they require many labeled samples when trained in a supervised setting. The costs of annotating data manually can render the use of DCNNs infeasible. We present a novel framework called RenderGAN that can generate large amounts of realistic, labeled images by combining a 3D model and the Generative Adversarial Network framework. In our approach, image augmentations (e.g. lighting, background, and detail) are learned from unlabeled data such that the generated images are strikingly realistic while preserving the labels known from the 3D model. We apply the RenderGAN framework to generate images of barcode-like markers that are attached to honeybees. Training a DCNN on data generated by the RenderGAN yields considerably better performance than training it on various baselines.
1 INTRODUCTION
When an image is taken from a real world scene, many factors determine the final appearance: background, lighting, object shape, position and orientation of the object, the noise of the camera sensor, and more. In computer vision, high-level information such as class, shape, or pose is reconstructed from raw image data. Most real-world applications require the reconstruction to be invariant to noise, background, and lighting changes.
In recent years, deep convolutional neural networks (DCNNs) advanced to the state of the art in many computer vision tasks (Krizhevsky et al., 2012; He et al., 2015; Razavian et al., 2014). More training data usually increases the performance of DCNNs. While image data is mostly abundant, labels for supervised training must often be created manually – a time-consuming and tedious activity. For complex annotations such as human joint angles, camera viewpoint or image segmentation, the costs of labeling can be prohibitive.
In this paper, we propose a method to drastically reduce the costs of labeling such that we can train a model to predict even complex sets of labels. We present a generative model that can sample from the joint distribution of labels and data. The training procedure of our model does not require any manual labeling. We show that the generated data is of high quality and can be used to train a model in a supervised setting, i.e. a model that maps from real samples to labels, without using any manually labeled samples.
We propose two modifications to the recently introduced GAN framework (Goodfellow et al., 2014). First, a simple 3D model is embedded into the generator network to produce samples from corresponding input labels. Second, the generator learns to add missing image characteristics to the model output using a number of parameterized augmentation functions. In the adversarial training we leverage large amounts of unlabeled image data to learn the particular form of blur, lighting, background and image detail. By constraining the augmentation functions we ensure that the resulting image still represents the given set of labels. The resulting images are hard to distinguish from real samples and can be used to train a DCNN to predict the labels from real input data.
The RenderGAN framework was developed to solve the scarcity of labeled data in the BeesBook project (Wario et al., 2015) in which we analyze the social behavior of honeybees. A barcode-like marker is attached to the honeybees’ backs for identification (see Fig. 1). Annotating this data is tedious, and therefore only a limited amount of labeled data exists. A 3D model (see the upper row of Fig. 2) generates a simple image of the tag based on position, orientation, and bit configuration. The RenderGAN then learns from unlabeled data to add lighting, background, and image details.
Training a DCNN on data generated by the RenderGAN yields considerably better performance compared to various baselines. We furthermore include a previously used computer vision pipeline in the evaluation. The networks’ detections are used as feature to track the honeybees over time. When we use detections from the DCNN instead of the computer vision pipeline, the accuracy of assigning the true id increases from 55% to 96%.
Our contributions are as follows. We present an extension of the GAN framework that allows to sample from the joint distribution of data and labels. The generated samples are nearly indistinguishable from real data for a human observer and can be used to train a DCNN end-to-end to classify real samples. In a real-world use case, our approach significantly outperforms several baselines. Our approach requires no manual labeling. The simple 3D model is the only form of supervision.
2 RELATED WORK
There exists multiple approaches to reduce the costs associated with labeling.
A common approach to deal with limited amount of labels is data augmentation (Goodfellow et al., 2016, Chapter 7.4). Translation, noise, and other deformations can often be applied without changing the labels, thereby effectively increasing the number of training samples and reducing overfitting.
DCNNs learn a hierarchy of features – many of which are applicable to related domains (Yosinski et al., 2014). Therefore, a common technique is to pre-train a model on a larger dataset such as ImageNet (Deng et al., 2009) and then fine-tune its parameters to the task at hand (Girshick et al., 2014; Long et al., 2015; Razavian et al., 2014). This technique only works in cases where a large enough related dataset exists. Furthermore, labeling enough data for fine-tuning might still be costly.
If a basic model of the data exists (for example, a 3D model of the human body), it can be used to generate labeled data. Peng et al. (2015) generated training data for a DCNN with 3D-CAD models.
Su et al. (2015) used 3D-CAD models from large online repositories to generate large amounts of training images for the viewpoint estimation task on the PASCAL 3D+ dataset (Xiang et al., 2014). Massa et al. (2015) are matching natural images to 3D-CAD models with features extracted from a DCNN. Richter et al. (2016) and Ros et al. (2016) used 3D game engines to collect labeled data for image segmentation. However, the explicit modeling of the image acquisition physics (scene lighting, reflections, lense distortions, sensor noise, etc.) is cumbersome and might still not be able to fully reproduce the particularities of the imaging process such as unstructured background or object specific noise. Training a DCNN on generated data that misses certain features will result in overfitting and poor performance on the real data.
Generative Adversarial Networks (GAN) (see Fig. 3) can learn to generate high-quality samples (Goodfellow et al., 2014), i.e. sample from the data distribution p(x). Denton et al. (2015) synthesized images with a GAN on the CIFAR dataset (Krizhevsky, 2009), which were hard for humans to distinguish from real images. While a GAN implicitly learns a meaningful latent embedding of the data (Radford et al., 2015), there is no simple relationship between the latent dimensions and the labels of interest. Therefore, high-level information can’t be inferred from generated samples. cGANs are an extension of GANs to sample from a conditional distribution given some labels, i.e. p(x|l). However, training cGANs requires a labeled dataset. Springenberg (2015) showed that GANs can be used in a semi-supervised setting but restricted their analysis to categorical labels. Wang & Gupta (2016) trained two separate GANs, one to model the object normals and another one for the texture conditioned on the normals. As they rely on conditional GANs, they need large amounts of labeled data. Chen et al. (2016) used an information theoretic to disentangle the representation. They decomposed the representation into a structured and unstructured part. And successfully related on a qualitative level the structured part to high-level concepts such as camera viewpoint or hair style. However, explicitly controlling the relationship between the latent space and generated samples without using labeled data is an open problem, i.e. sampling from p(x, l) without requiring labels for training.
3 RENDERGAN
Most supervised learning tasks can be modeled as a regression problem, i.e. approximating a function f̂ : Rn 7→ L that maps from data space R to label space L. We consider f̂ to be the best available function on this particular task. Analogous to ground truth data, one could call f̂ the ground truth function.
In the RenderGAN framework, we aim to solve the inverse problem to this regression task: generate data given the labels. This is achieved by embedding a simple 3D model into the generator of a GAN. The samples generated by the simple model must correspond to the given labels but may lack many factors of the real data such as background or lightning. Through a cascade of augmentation functions, the generator can adapt the images from the 3D model to match the real data.
We formalize image augmentation as a function φ(x, d), which modifies the image x based on the augmentation parameter d (a tensor of any rank). The augmentation must preserve the labels of the image x. Therefore, it must hold for all images x and all augmentations parameters d:
f̂ (φ(x, d)) = f̂(x) (1)
The augmentation function must furthermore be differentiable w.r.t. x and d as the gradient will be back-propagated through φ to the generator. Image augmentations such as lighting, surrounding, and noise do preserve the labels and fit this definition. We will provide appropriate definitions of φ for the mentioned augmentations in the following section.
If appropriate augmentation functions are found that can model the missing factors and are differentiable, we can use the GAN framework to find parameters that result in realistic output images. Multiple augmentation functions can be combined to perform a more complex augmentation. Here, we will consider multiple augmentation functions applied sequentially, i.e. we have k augmentation functions φi and k corresponding outputs Gi from the generator. The output of the previous augmentation function is the input to the next one. Thus, we can write the generator given some labels l as:
g(z, l) = φk(φk−1(. . . φ0(M(l), G0(z)) . . . , Gk−1(z)), Gk(z)) (2)
where M is the 3D model. We can furthermore learn the label distribution with the generator. As the discriminator loss must be backpropagated through the 3D model M, it must be differentiable. This can be achieved by emulating the 3D model with a neural network (Dosovitskiy et al., 2015). The resulting generator g(z) can be written as (see Fig. 4 for a visual interpretation):
g(z) = φk(φk−1(. . . φ0(M(Gl(z)), G0(z)) . . . , Gk−1(z)), Gk(z)) (3)
As any differentiable function approximator can be employed in the GAN framework, the theoretical properties still hold. The training is carried out as in the conventional GAN framework. In a real application, the augmentation functions might restrict the generator from converging to the data distribution.
If the training converges, we can collect generated realistic data with g(z) and the high-level information captured in the 3D model with Gl(z). We can now train a supervised learning algorithm on the labeled generated data (Gl (z) , g (z)) and solve the regression task of approximating f̂ without depending on manual labels.
4 APPLICATION TO THE BEESBOOK PROJECT
In the BeesBook project, we aim to understand the complex social behavior of honey bees. For identification, a tag with a binary code is attached to the back of the bees.
The orientations in space, position, and bits of the tags are required to track the bees over time. Decoding this information is not trivial: the bees are oriented in any direction in space. The tag might be partially occluded. Moreover, spotlights on the tag can sometimes even confuse humans. A previously used computer vision pipeline did not perform reliably. Although we invested a substantial amount of time on labeling, a DCNN trained on this data did not perform significantly better (see Sec. 3). We therefore wanted to synthesize labeled images which are realistic enough to train an improved decoder network.
Following the idea outlined in section 3, we created a simple 3D model of a bee marker. The 3D model comprises a mesh which represents the structure of the marker and a simple camera model to project the mesh to the image plane. The model is parameterized by its position, its pitch, yaw and roll, and its ID. Given a parameter set, we obtain a marker image, a background segmentation mask
and a depth map. The generated images lack many important factors: blur, lighting, background, and image detail (see Fig. 2). A DCNN trained on this data does not generalize well (see Sec. 5).
Over the last years we collected a large amount of unlabeled image data. We successfully augmented the 3D model using this dataset, as described below.
We trained a neural network to emulate the 3D model. Its outputs are indistinguishable from the images of the 3D model. The discriminator error can now be backpropagated through the 3D model which allows the generator to also learn the distributions of positions and orientations of the bee marker. The IDs are sampled uniformly during training. The weights of the 3D model network are fixed during the RenderGAN training.
We apply different augmentation functions that account for blur, lighting, background, and image detail. The output of the 3D model and of each augmentation function is of shape (64, 64) and in the range [−1, 1]. In Fig. 5, the structure of the generator is shown. Blurriness: The 3D model produces hard edges, but the images of the real tags show a broad range of blur. The generator produces a scalar α ∈ [0, 1] per image that controls the blur.
φblur(x, α) = (1− α) (x− bσ (x)) + bσ(x) (4)
where bσ(x) = x ∗ kσ denotes convolving the image x with a Gaussian kernel kσ of scale σ. The implementation of the blur function is inspired by Laplacian pyramids (Burt & Adelson, 1983). As required for augmentation functions, the labels are preserved, because we limit the maximum amount of blur by picking σ = 2. φblur is also differentiable w.r.t the inputs α and x.
Lighting of the tag: The images from the 3D model are binary. In real images, tags exhibit different shades of gray. We model the lighting by a smooth scaling and shifting of the pixel intensities. The generator provides three outputs for the lighting: scaling of black parts sb, scaling of white parts sw and a shift t. All outputs have the same dimensions as the image x. An important invariant is that the black bits of the tag must stay darker than the white bits. Otherwise, a bit could flip, and the label would change. By restricting the scaling sw and sb to be between 0.10 and 1, we ensure that this invariant holds. The lighting is locally corrolated and should cause smooth changes in the image. Hence, Gaussian blur b(x) is applied to sb, sw, and t.
φlighting(x, sw, sb, t) = x · b(sw) ·W (x) + x · b(sb) · (1−W (x)) + b(t) (5)
The segmentation mask W (x) is one for white parts and zero for the black part of the image. As the intensity of the input is distributed around -1 and 1, we can use thresholding to differentiate between black and white parts.
Background: The background augmentation can change the background pixels arbitrarily. A segmentation mask Bx marks the background pixels of the image x which are replaced by the pixels from the generated image d.
φbg(x, d) = x · (1−Bx) + d ·Bx (6)
The 3D model provides the segmentation mask. As φbg can only change background pixels, the labels remain unchanged.
Details: In this stage, the generator can add small details to the whole image including the tag. The output of the generator d is passed through a high-pass filter to ensure that the added details are small enough not to flip a bit. Furthermore, d is restricted to be in [−2, 2] to make sure the generator cannot avoid the highpass filter by producing huge values. With the range [−2, 2], the generator has the possibility to change black pixels to white, which is needed to model spotlights.
φdetail(x, d) = x+ highpass(d) (7) The high-pass is implemented by taking the difference between the image and a blurred version of the image (σ = 3.5). As the spotlights on the tags are only a little smaller than the bits, we increase its slope after the cutoff frequency by repeating the high-pass filter three times.
The image augmentations are applied in the order as listed above: φdetail ◦φbackground ◦φlighting ◦ φblur. Please note that there exist parameters to the augmentation functions that could change the labels. As long as it is guaranteed that such augmentations will result in unrealistic looking images, the generator network will learn to avoid them. For example, even though the detail augmentation could be used to add high-frequency noise to obscure the tag, this artifact would be detected by the discriminator.
Architecture of the generator: The generator network has to produce outputs for each augmentation function. We will outline only the most important parts. See our code available online for all the details of the networks1. The generator starts with a small network consisting of dense layers, which predicts the parameters for the 3D model (position, orientations). The output of another dense layer is reshaped and used as starting block for a chain of convolution and upsampling layers. We found it advantageous to merge a depth map of the 3D model into the generator as especially the lighting depends on the orientation of the tag in space. The input to the blur augmentation is predicted by reducing an intermediate convolutional feature map to a single scalar. An additional network is branched off to predict the input to the lighting augmentation. For the background generation, the output of the lighting network is merged back into the main generator network together with the actual image from the 3D model.
For the discriminator architecture, we mostly rely on the architecture given by Radford et al. (2015), but doubled the number of convolutional layers and added a final dense layer. This change improved the quality of the generated images.
Clip layer: Some of the augmentation parameters have to be restricted to a range of values to ensure that the labels remain valid. The training did not converge when using functions like tanh or sigmoid due to vanishing gradients. We are using a combination of clipping and activity regularization to keep the output in a given interval [a, b]. If the input x is out of bounds, it is clipped and a regularization loss r depending on the distance between x and the appropriate bound is added.
r(x) =  γ||x− a||1 if x < a 0 if a ≤ x ≤ b γ||x− b||1 if x > b
(8)
f(x) = min(max(a, x), b) (9) With the scalar γ, the weight of the loss can be adapted. For us γ = 15 worked well. If γ is chosen too small, the regularization loss might not be big enough to move the output of the previous layer towards the interval [a, b]. During training, we observe that the loss decreases to a small value but never vanishes.
Training: We train generator and discriminator as in the normal GAN setting. We use 2.4M unlabeled images of tags to train the RenderGAN. We use Adam (Kingma & Ba, 2014) as an optimizer with a starting learning rate of 0.0002, which we reduce in epoch 200, 250, and 300 by a factor of 0.25. In Fig. 6b the training loss of the GAN is shown. The GAN does not converge to the point where the discriminator can no longer separate generated from real samples. The augmentation functions might restrict the generator too much such that it cannot model certain properties. Nevertheless, it is hard for a human to distinguish the generated from real images. In some cases, the
1https://github.com/berleon/deepdecoder
generator creates unrealistic high-frequencies artifacts. The discriminator unfailingly assigns a low score to theses images. We can therefore discard them for the training of the supervised algorithm. More generated images are shown in Appendix A. In Fig. 7, we show random points in the latent space, while fixing the tag parameters. The generator indeed learned to model the various lighting conditions, noise intensities, and backgrounds.
5 RESULTS
We constructed the RenderGAN to generate labeled data. But does a DCNN trained with the RenderGAN data perform better than one trained on the limited amounts of real data? And are learned augmentations indeed needed or do simple hand-designed augmentation achieve the same result? The following paragraphs describe the different datasets used in the evaluation. We focus on the performance of a DCNN on the generated data. Thus, we do not compare our method to conventional GANs as those do not provide labels and are generally hard to evaluate.
Data from the RenderGAN: We generate 5 million tags with the RenderGAN framework. Due to the abundance, one training sample is only used twice during training. It is not further augmented.
Real Data: The labels of the real data are extracted from ground truth data that was originally collected to evaluate bee trajectories. This ground truth data contains the path and id of each bee over multiple consecutive frames. Data from five different time spans was annotated – in total 66K tags. As the data is correlated in time (same ids, similar lighting conditions), we assign the data from one time span completely to either the train or test set. The data from three time spans forms the train set (40K). The test set (26K) contains data from the remaining two time spans. The ground truth data lacks the orientation of the tags, which is therefore omitted for this evaluation. Due to the smaller
size of the real training set, we augment it with random translation, rotation, shear transformation, histogram scaling, and noise (see Appendix C for exact parameters).
RenderGAN + Real: We also train a DCNN on generated and real data which is mixed at a 50:50 ratio.
Handmade augmentations: We tried to emulate the augmentations learned by the RenderGAN by hand. For example, we generate the background by an image pyramid where the pixel intensities are drawn randomly. We model all effects, i.e. blur, lighting, background, noise and spotlights (see Appendix B for details on their implementation). We apply the handmade augmentation to different learned representations of the RenderGAN, e.g. we use the learned lighting representation and add the remaining effects such as background and noise with handmade augmentations (HM LI). See Table 1 for the different combinations of learned representations and hand designed augmentations.
Computer vision pipeline CV : The previously used computer vision pipeline (Wario et al., 2015) is based on manual feature extraction. For example, a modified Hough transformation to find ellipses. The MHD obtained by this model is only a rough estimate given that the computer vision pipeline had to be evaluated and fine-tuned on the same data set due to label scarcity.
Training setup: An epoch consists of 1000 batches á 128 samples. We use early stopping to select the best parameters of the networks. For the training with generated data, we use the real training set as the validation set. When training on real data, the test set is also used for validation. We could alternatively reduce the real training set further and form an extra validation set, but this would harm the performance of the DCNN trained on the real data. We use the 34-layer ResNet architecture (He et al., 2015) but start with 16 feature maps instead of 64. The DCNNs are evaluated on the mean Hamming distance (MHD) i.e. the expected value of bits decoded wrong. Human experts can decode tags with a MHD of around 0.23.
Results: In Table 2, we present the results of the evaluation. The training losses of the networks are plotted in Fig. 8. The model trained with the data generated by the RenderGAN has an MHD of 0.424. The performance can furthermore be slightly improved by combining the generated with real data. The small gap in performance when adding real data is a further indicator of the quality of the generated samples.
If we use predictions from this DCNN instead of the computer vision pipeline, the accuracy of the tracking improves from 55% of the ids assigned correctly to 96%. At this quality, it is possible to analyze the social behavior of the honeybees reliably.
Compared to the handmade augmentations (HM 3D), data from the RenderGAN leads to considerably better performance. The large gap in performance between the HM 3D and HM LI data highlights the importance of the learned lighting augmentation.
6 DISCUSSION
We proposed a novel extension to the GAN framework that is capable of rendering samples from a basic 3D model more realistic. Compared to computer graphics pipelines, the RenderGAN can learn complex effects from unlabeled data that would be otherwise hard to model with explicit rules.
Contrary to conventional GANs, the generator provides explicit information about the synthesized images, which can be used as labels for a supervised algorithm. The training of the RenderGAN requires no labels.
We showed an application of the RenderGAN framework to the BeesBook project, in which the generator adds blur, lighting, background, and details to images from a basic 3D model. The generated data looks strikingly real and includes fine details such as spotlights, compression artifacts, and sensor noise.
In contrast to previous work that applied 3D models to produce training samples for DCNNs (Su et al., 2015; Richter et al., 2016; Ros et al., 2016), we were able to train a DCNN from scratch with only generated data that still generalizes to unseen real data.
While some work is required to adapt the RenderGAN to a specific domain, once set up, arbitrary amounts of labeled data can be acquired cheaply, even if the data distribution changes. For example, if the tag design changes to include more bits, small adaptions to the 3D model’s source code and eventually the hyperparameters of the augmentation functions would be sufficient. However, if we had labeled the data by hand, then we would have to annotate data again.
While the proposed augmentations represent common image characteristics, a disadvantage of the RenderGAN framework is that these augmentation functions must be carefully customized for the application at hand to ensure that high-level information is preserved. Furthermore, a suitable 3D model must be available.
7 FUTURE WORK
For future work, it would be interesting to see the RenderGAN framework used on other tasks where basic 3D models exist e.g. human faces, pose estimation, or viewpoint prediction. In this context, one could come up with different augmentation functions e.g. colorization, affine transformations, or diffeomorphism. The RenderGAN could be especially valuable to domains where pre-trained models are not available or when the annotations are very complex. Another direction of future work might be to extend the RenderGAN framework to other fields. For example, in speech synthesis, one could use an existing software as a basic model and improve the realism of the output with a similar approach as in the RenderGAN framework.
B HANDMADE AUGMENTATIONS
We constructed augmentations for blur, lighting, background, noise and spotlights manually. For synthesizing lighting, background and noise, we use image pyramids, i.e. a set of images L0, . . . , L6 of size (2i × 2i) for 0 ≤ i ≤ 6. Each level Li in the pyramid is weighted by a scalar ωi. Each pixel of the different level Li is drawn from N (0, 1). The generated image I6 is given by:
I0 = ω0L0 (10) Ii = ωiLi + upscale(Ii−1) (11)
, where upscale doubles the image dimensions. The pyramid enables us to generate random images while controlling their frequency domain by weighting the pyramid levels appropriately.
• Blur: Gaussian blur with randomly sampled scale. • Lighting: Similar as in the RenderGAN. Here, the scaling of the white and black parts
and shifting is constructed with image pyramids. • Background: image pyramids with the lower levels weight more. • Noise: image pyramids with only the last two layer. • Spotlights: overlay with possible multiple 2D Gauss function with a random position on
the tag and random covariance.
We selected all parameters manually by comparing the generated to real images. However, using slightly more unrealistic images resulted in better performance of the DCNN trained with the HM 3D data. The parameters of the handmade augmentations can be found online in our source code repository.
C AUGMENTATIONS OF THE REAL DATA
We scale and shift the pixel intensities randomly, i.e. sI+t, where I is the image and s, t are scalars. The noise is sampled for each pixel fromN (0, ), where ∼ max(0, N (µn, σn)) is drawn for each image separately. Different affine transformations (rotation, scale, translation, and shear) are used.
D TRAINING SAMPLES

The submission proposes an interesting way to match synthetic data to real data in a GAN type architecture.
The main novelty are parametric modules that emulate different transformations and artefact that allow to match the natural appearance.
several points were raised during the discussion:
1. the proposed method is more model driven that previous GAN models. But does it pay off? how would a traditional GAN approach perform? The mentioned effects like blur, lighting and background could also potentially be modelled by upsamling network that directly predicts the image. I would assume that blur and lighting can be modelled by convolutions. transformations to some extend by convolutions - or spatial transformer networks.
The answers of the authors only partially addresses the point. The key proposal of the submission seems parameterised modules that can be trained to match the real data distribution. but it remains unclear why not a more generic parameterisation can also do the job. E.g. a neural network - as done in regular GANs. The benefit of introducing a stronger model is unclear. Using a render engine to generate the initial sample appearance if of limited novelty.
2. how does it compare to traditional data augmentation techniques, e.g. noise, dropout, transformations. you are linking to keras code - where data augmentation is readily available and could be tested (ImageDataGenerator)
The authors reply that plenty of such augmentation was used and more details are going to be provided in the appendix. it would have been appreciated if such information was directly included in the revision - so that the procedure could be directly checked. right now - this remains a point of uncertainty.
3. How do the different stages (\phis) effect performance? which are the most important ones?
The authors do evaluate the effect of hand tuning the transformation stages vs. learning them. it would be great to also include results of including/excluding stages completely - and also reporting how much the initial jittering of the data helps.
While there is an interesting idea of (limited) novelty to the paper, there are some concerns about evalations and comparisons as outlined above. In addition, only success on a single dataset/task is shown. Yet the task is interesting and seems challenging. Overall, this remains makes only a weak recommendation for acceptance.