Deploying deep neural networks on mobile devices is a challenging task due to computation complexity and memory intensity. Existing works solve this problem by reducing model size using weight compression methods based on dimension reduction (i.e., SVD, Tucker decomposition and Quantization). However, the execution speed of these compressed models are still far below the real-time processing requirement of mobile services. To address this limitation, we propose a novel acceleration framework: DeepRebirth by exploring the deep learning model parameter sparsity through merging the parameter-free layers with their neighbor convolution layers to a single dense layer. The design of DeepRebirth is motivated by the key observation: some layers (i.e., normalization and pooling) in deep learning models actually consume a large portion of computational time even few learned parameters are involved, and acceleration of these layers has the potential to improve the processing speed significantly. Essentially, the functionality of several merged layers is replaced by the new dense layer – rebirth layer in DeepRebirth. In order to preserve the same functionality, the rebirth layer model parameters are re-trained to be functionality equivalent to the original several merged layers. The extensive experiments performed on ImageNet using several popular mobile devices demonstrate that DeepRebirth is not only providing huge speed-up in model deployment and significant memory saving but also maintaining the model accuracy, i.e., 3x-5x speed-up and energy saving on GoogLeNet with only 0.4% accuracy drop on top-5 categorization in ImageNet. Further, by combining with other model compression techniques, DeepRebirth offers an average of 65ms model forwarding time on single image using Samsung Galaxy S6 with only 2.4% accuracy drop. In addition, 2.5x run-time memory saving is achieved with rebirth layers.
1 INTRODUCTION
Recent years have witnessed the breakthrough of deep learning techniques for image classification and object recognition. Mobile device becomes more and more popular due to its convenient mobiles services provided for end users. More and more mobile applications require deep learning techniques to provide accurate, intelligent and effective services. However, the execution speed of the deep learning model on mobile devices becomes a bottleneck for many applications due to the large model size, deep network structure and complicated model parameters, which hinders the real-time deployment. However, if deep learning service is only provided at cloud side, transmis-
sion of images though internet may compromise image owners’ privacy and is also limited by the availability of the Internet.
Running deep learning models efficiently on mobile CPUs is a highly intriguing feature due to many reasons: (1) CPU is available for all mobile devices, even phones released many years ago; (2) powerful CUDA-enabled GPUs are generally not available on (compact) mobile devices; (3) though a large majority of mobile devices are equipped with mobile GPUs, the speed-up achieved on the mobile GPUs is quite limited when compared to CPU sh1r0 et al. (2015), not to mention the complexity caused by different mobile GPU architectures; (4) major deep learning frameworks such as Caffe Jia et al. (2014) and Tensorflow Abadi et al. (2015) only support CPU implementation on mobile devices currently, and therefore an efficient CPU-friendly model is highly desirable.
However, most of current mobile CPUs cannot meet the needs of deep learning model deployment because it takes much longer time and higher energy cost to process an image using pre-trained deep learning models. For example, it takes more than 651ms to recognize an image using GoogleNet on Samsung S5 (Table 4) with 984mJ energy costs (Table 5). Therefore a question that naturally follows is: can we develop an efficient deep learning acceleration framework to facilitate deployment of deep learning service on mobile device?
This problem is challenging due to the fact that the practical solution is highly desirable to support different practical scenarios by addressing the following challenges (C1–C3).
C1: Minimum accuracy loss. The solution is expected to provide minimum accuracy loss while provide great speed-up.
C2: Leveraging existing trained deep framework. In order to provide the best deep learning service, the mechanism is designed to taking advantage of existing state-of-the-art deep learning architectures (e.g., GoogLeNet and ResNet) instead of training from scratch.
C3: Supporting different deep learning architecture components. The proposed technique should provide generic framework that can be applied to these popular deep learning models that may consist of different types of layers. In general, all neural network layers can be grouped into two categories: tensor layer and non-tensor layer based on whether the layer contains tensor-type parameters. For example, fully connected layer and the convolution layer are both tensor-layers since they contain 2-d and 4-d tensor-type weight parameters, respectively. Pooling layer and LRN layer are both non-tensor layers because they do not contain any high-order tensor-type weight parameters1. Therefore, the framework is expected to support both tensor and non-tensor layers optimization.
However, the current solutions for deep learning model acceleration are still quite limited in addressing these challenges. The main goal of works ( Han et al. (2016b), Li (2013); Kim et al. (2015); Jiaxiang Wu & Cheng (2016)) is to reduce the model size by approximating the tensor-type layers using low rank approximation and vector quantization techniques. While they can provide some acceleration for only fully-connected layers (used in AlexNet, VGGNet), the application scenarios of these methods are very limited and ineffective because modern deep learning architectures (e.g., Inception and ResNet) have removed large fully-connected layers. Moreover, for non-tensor layers (e.g., normalization and pooling layers) that are generally used for speeding up the network training and obtaining better generalization performance, none works, to the best of our knowledge, have discussed how to accelerate the execution process.
To bridge these gaps, this paper proposes DeepRebirth, a new deep learning model acceleration framework by exploring the sparsity of deep neural network layers to accelerate both non-tensor layers and tensor layers from two types of rebirth: streaming merging and branch merging. In streaming merging, the new tensor layers are generated by merging non-tensor layers with its neighboring sparse tensor layers in the feed-forward structure as illustrated in Figure 2, while in branch merging, the new tensor layers are created by fusing non-tensor banches with the sparse tensor branches (at the same level) as shown in Figure 3, i.e., the inception module in GoogLeNet ( Szegedy et al. (2014)). The design of DeepRebirth is guided by the key observation:
Non-tensor layers are the major obstacles for real-time mobile CPU execution (Section 2).
Then reducing the execution time on non-tensor layers can greatly reduce the overall model forwarding time. In order to reduce the execution time, both streaming merging and branch merging
1Other examples of non-tensor layers include dropout layer, normalization layer, softmax layer, etc.
are applied to merge non-tensor layers into tensor layers. Overall, reducing the execution time on non-tensor layers can greatly reduce the model forwarding time given the fact that tensor-layer has been optimized to the minimum as suggested by ( Han et al. (2016b), Kim et al. (2015)). Ideally, we can combine both non-tensor and tensor layer optimization together and further reduce latency as well as the model size. To summarize, this paper makes the following contributions.
· Our approach is the first work that optimizes non-tensor layers and significantly accelerates a deep learning model on CPUs while reducing the required runtime-memory since there are less layers in the reconstructed deep learning model2.
· To address the challenges of (C1–C3), we perform both streaming merging and branch merging based on the original structure of old layers while the new tensor layers are generated by merging non-tensor layers with its neighboring sparse tensor layers vertically and horizontally.
· As demonstrated in the experiment, our approach has obtained the state-of-the-art speeding up on popular deep learning models with negligible accuracy loss. Our proposed method enables GoogLeNet to achieve 3x-5x speed-up for processing a single image with only 0.4% drop on Top-5 accuracy on ImageNet without any weights compression method. By further applying model compression techniques, we achieve around 65 ms for processing a single image with Top-5 accuracy of 86.5%. Furthermore, we show that our methods work for state-of-the-art non-tensor layers, e.g., batch normalization, in very deep neural network models such as ResNet He et al. (2015).
2 WHAT IS THE IMPACT OF NON-TENSOR LAYERS ON SPEED?
Experimental Settings To give a better understanding of the neural network latency, we evaluate the time cost of different types of layers within a given network. We measure their latency by using the time percentage measurement where larger value indicates longer time 3. Our experiment is carried on different processors including Intel x86 CPU, Arm CPU and Titan X GPU. Along with different processors, we also use different state-of-the-art networks to evaluate. These networks
2Tensor weights decomposition method such as Tucker Decomposition effectively reduces the model size (i.e., the number of learned weights) and thus reduce the storage cost on hard drive. However, since the decomposition methods increase the number of layers of the model, the actual runtime-memory (RAM) cost (which is much more scarce resource than hard drive storage) can be even larger than the model before decomposition.
3The accumulated percentage for a given network is 100%.
include AlexNet (Figure 1a, Krizhevsky et al.), GoogLeNet(Figure 1b, Szegedy et al. (2014)) and ResNet(Figure 1c, He et al. (2015)). We list the results in Figure 1 and Table 1.
Observations and Insights As demonstrated in the results, for classical deep models (e.g., AlexNet), among the non-tensor layers, “LRN” and “Pooling” layers are the major obstacles that slow-down the model execution. ResNet-50 has abandoned the “LRN” layers by introducing the batch normalization layer, but the findings remain valid as it takes up more than 25% of the time on ARM CPU and more than 40% on Intel x86 CPU (in Caffe ( Jia et al. (2014)), it was decomposed into a “BatchNorm” layer followed by a “Scale” layer as shown in Figure 1c). The time fraction spent over such layers ranges from 22.37% to 62.03%. Among different types of processors, non-tensor layers have the largest impact on Intel x86 CPUs, and more specifically 62.03% of the computing time. On the other hand, though non-tensor layers do not affect the mainstream ARM CPUs, on average they still cost about 1/3 of the computing time. All these numbers confirm our intuition: there is a great potential to accelerate the model by optimizing those non-tensor layers.
3 DEEPREBIRTH
This section covers the design of DeepRebirth in three aspects: streaming merging, branching merging and adapting DeepRebirth to the whole model.
In general deep learning models, the probability distribution of the dataset can be represented by a large, very sparse deep neural network that is constructed layer after layer. From analyzing the correlations of the current layer and preceding layers (or parallel layers), we can merge the highly correlated layers and substitute it as a new “rebirth” layer. This process is similar to viewing the Inception model as a logical culmination as suggested by Arora et al. (2013).
3.1 STREAMLINE MERGING
For deep network architecture with streamline layer connections, in order to accelerate the execution, we first identify the layers that have large latency but also have potentials to be merged or processed. The merging design is motivated by the following two key observations.
• Non-tensor layers are usually following a tensor layer such as convolution layer as shown in Figure 2.
• Several consecutive layers can be viewed as a blackbox for non-linear transformations, and therefore this can be replaced by a new tensor-layer by learning the parameters to approximate the functionality of original several layers. An example is shown in Figure 2.
Method The streamline merging regenerates a new tensor layer (i.e., rebirth layer) by merging nontensor layers with its bottom tensor units in the feed-forward structure. After layer-wise regeneration, we retrain the deep neural network model by fine-tuning the parameters of the new generated layers. There are two streamline merging operations in the proposed scheme. The choice of merging operation is depending on the type of non-tensor layers.
• Merging Pooling Layer: The pooling layer down-samples feature maps learned from previous layers. Therefore, to merge a pooling layer to a convolution layer, we remove the pooling layer and set the stride value of the “merged” convolution layer as the product of the stride values for both the original pooling layer and the convolution layer. With a larger stride value for the new “merged” convolution layer, it further reduces the computation required for executing the new model.
• Merging Non-Pooling Layer: For non-pooling layers such as LRN and batch normalization, we directly prune those layers from the original deep neural network.
Example Figure 2 illustrates how the optimization works using streamline merging. This is one representative part in GoogLeNet where the convolution layer conv2/3 × 3 is followed by a LRN layer conv2/norm2 and a pooling layer poo2/3 × 3 s2 (The ReLU layer which has negligible latency is retained to keep accuracy). Before merging, the 2 non-tensor layers without a single learned parameter weight take even more time than running the convolution layer. After merging
process them to generate a new rebirth convolution layer conv2/3× 3 merge, the time spent on the rebirth layer is greatly reduced compare to the original layers.
3.2 BRANCH MERGING
The design of branch merging is motivated by the following key observation. Given the fact that non-tensor layer requires more time on computation, if we can learn new tensor layers by fusing non-tensor layers with the tensor units at the same layer level, then the the execution time will be decreased.
Example One representative unit is the inception module in GoogLeNet. For example as illustrated in Figure 3, layer “inception 3a” of GoogLeNet has 4 branches: 3 convolution branches take feature maps from the bottom layer at various scales (1× 1, 3× 3 and 5× 5) and 1 additional 3× 3 pooling branch Szegedy et al. (2014). The output feature maps of each branch are concatenated as input for the following top layer.
Method For deep network architecture with parallel branches, the output of each branch constitutes part of the feature maps as the input for the next layer. We identify non-tensor branches that have large latency (e.g., the pooling branch in Figure 3). Similar to streamline merging, if we can use a faster tensor branch to simulate the function of the non-tensor branch by relearning its parameters, we can achieve clear speed-up.
To merge a non-tensor branch into a tensor branch, we re-create a new tensor layer (i.e., rebirth layer) by fusing the non-tensor branch and a tensor unit with relatively small latency to output the feature maps that were originally generated by the non-tensor branch. If the non-tensor branch has a kernel size larger than 1× 1 (e.g., the 3× 3 pooling branch in Figure 3), the picked tensor branch’s kernel size should be at least the size of the non-tensor branch. As shown in this figure, we re-learn a new tensor layer “inception 3a” by merging the 3× 3 pooling branch with the 5× 5 convolution branch at the same level, and the number of feature maps obtained by the 5 × 5 convolution is increased from 32 to 64.
Explore Sparsity for Tensor-layer Branch Reducing and Merging
• Reducing: Current deep neural networks usually include convolution branches with 1 × 1 convolution layers (e.g., inception 3a/3x3 reduce in Figure 3) aiming to reduce feature maps channels. This unit will be processed by a following convolution layer with larger kernel size. For greater speed-up, we further reduce the number of feature maps generated
by the 1×1 “reducer”. For layer inception 3a/3x3 reduce, we reduce the number of output feature maps from 96 to 48.
• Merging: A convolution branch with a smaller kernel size can be merged to a convolution branch with a larger kernel size. The method is similar to the merging of non-tensor layers. To keep other layers’ structures in network unchanged, we remove the small-kernel convolution branch and increase the number of feature maps generated by the large-kernel convolution layers. For examples, for layer inception 3a/3x3 reduce, we remove the 1× 1 convolution branch and increase the number of feature maps generated by the 3× 3 convolution from 128 to 196.
3.3 ADAPTING DEEPREBIRTH TO OVERALL MODEL
The new generated layer (i.e., rebirth layer) is required to learn the new parameters using finetuning as discussed in Yosinski et al. (2014); Razavian et al. (2014). We use standard initialization methods to (e.g., Xavier Glorot & Bengio (2010) initialization) to initialize the parameters in the new layer while keeping the weights of other layers unchanged. In our optimization procedure, we set the learning rate of the new learning layers 10 times over those in other layers. The proposed optimization scheme is applied from the bottom layer to the top layer. It is also possible to learn multiple rebirth layers at the same time (we merge and fine-tune 3 sequential inception layers 4b-4d together for GoogLeNet) or merge layers in orders other than bottom-to-top.
4 EVALUATION
4.1 GOOGLENET
To evaluate the performance of DeepRebirth, we performed a comprehensive evaluation using ifferent optimization approaches on top of GoogLeNet. We use Caffe’s GoogLeNet implementation (i.e., bvlc googlenet) with its pre-trained model weights. Then we apply the proposed DeepRebirth optimization scheme to accelerate the running speed of GoogLeNet, which is denoted as “GoogLeNetMerge” (see structure in appendix). After non-tensor layer optimization (streamline and branch merging), we further apply tucker decomposition approach (Kim et al. (2015)) to reduce the model size (i.e., the number of learned weights) by 50%, represented as “GoogLeNet-Merge-Tucker”. In addition, we directly employ tucker decomposition method to compress original GoogLeNet. This
is indicated as “GoogLeNet-Tucker”. Thus, we have 4 models to compare, namely GoogLeNet, GoogLeNet-Merge, GoogLeNet-Tucker and GoogLeNet-Merge-Tucker.
4.1.1 ACCURACY
Since one of our major goals is to propose a new acceleration approach which can speed up the model running time with satisfied accuracy (in constrast to the original model), we list the accuracy changes along with the optimization steps conducted on ImageNet ILSVRC-2012 validation dataset as indicated in Table 2. During the whole optimization procedure of model training, we set the base learning rate for the re-generated layer as 0.01 (the rest layers are 0.001). We apply stochastic gradient descent training method (Bottou (2012)) to learn the parameters with a batch size of 32. During our training phase, we set 40,000 as the step size together with 0.1 set for gamma value and 0.9 for momentum parameter. At each step, the model generally converges at around 90,000 iterations (2 epochs).
The result indicates that the proposed method has almost negligible impact on the model accuracy, and the accuracy even increases at certain step (e.g., step 5). This indicates that “the newborn” layers perfectly simulate the functionalities of previous non-tensor layers before optimization. By applying tucker decomposition method on the merged model to reduce the weights by half (GoogLeNet-Merge-Tucker), we observer that there is a larger drop on accuracy (around 2%). However, directly applying tucker decomposition method (GoogLeNet-Tucker) to reduce the GoogLeNet weights to a half drops the top-5 accuracy to 85.7%. These results imply that our method performs reasonable well even after streamline and branch layer mergings.
4.1.2 SPEED-UP
To evaluate and compare the latency of different optimization approaches, we evaluate the the layerwise running speed on a Samsung Galaxy S5 smart phone which has an ARMv7 quad-core CPU @ 2.5 GHz and 2 GB RAM. We use Caffe’s integrated benchmark module to test the model forwarding time. Each test run includes 50 subtests with a random input. We try 10 test runs on each compared model and report the best test run in terms of forwarding time. During the whole experiment, we turn on phone to the airplane mode and close all other apps.
As is demonstrated in Table 3, we observe that for the best case scenario, GoogLeNet-Merge is 3x faster than GoogLeNet and for the worst case scenario, GoogLeNet takes around 950 ms for a single forwarding while GoogLeNet-Merge takes only around 250 ms, which is almost 4x speedup. This is because the original GoogLeNet model has too many small layers and this results in performance fluctuation. The same finding is also sharply observed in Kim et al. (2015) . The Tucker Decomposition method further reduces the computation for around 50% at the cost of around 2% accuracy loss. On the other hand, directly applying tucker decomposition on tensor layers doesn’t show any significant acceleration.
Not limited to mobile platform of Samsung Galaxy S5, we also apply the speed-up schemes on other popular processors. These mobile devices include (1) Moto E: a low-end mobile ARM CPU, (2) Samsung Galaxy S5: a middle-end mobile ARM CPU, (3) Samsung Galaxy S6: a high-end mobile ARM CPU, (4) Macbook Pro: an Intel x86 CPU, and (5) Titan X: a powerful server GPU. We demonstrate the experimental results in Table 4. The promising result indicates that the proposed method achieves significant speed-up on various types of CPUs. Even on the low-end mobile CPU (i.e., Moto E), around 200 ms model forwarding time is achieved by further applying tensor weights compression method. Finally, we compare the proposed approach with SqueezeNet (Iandola et al. (2016)) which is a state-of-the-art compressed CNN model. We are very excited to see that our optimization approach can obtain faster speed with higher accuracy compared to SqueezeNet(80% for Top-5)’s performance on all CPU platforms as listed in Table 4.
4.1.3 ENERGY, STORAGE AND RUNTIME-MEMORY COST
We measure the energy cost of each compared model using PowerTutor Android app (Zhang et al. (2010)) on Samsung Galaxy S5. The original GoogLeNet consumes almost 1 Joule per image while GoogLeNet-Merge consumes only 447 mJ. Applying tucker decomposition further reduces the energy cost to only 1/4 at 226 mJ .
When deploying to the mobile devices, we remove the loss1 and loss2 branches from the trained models so that the storage cost of each model is reduced by 24.33 MB. GoogLeNet-Merge which achieves significant speed-up does not save much storage cost compared to the original GoogLeNet model. However, for modern mobile devices, storage is not a scarce resource (e.g., Samsung Galaxy S5 has 16 GB or 32 GB storage), so a 20 MB deep learning model is “affordable” on mobile devices. Meanwhile, we can always perform the tensor weights compression method to further reduce the storage cost.
Another benefit of layer merging is run-time memory saving. The generated GoogLeNet-Merge model reduces the number of layers and consumes only 13.2 MB to process one image. This feature is also very useful for the cloud based deep learning service which can process a much larger batch at one run. As shown in table 5, one Titan X GPU can run a batch size of 882 with the GoogLeNetMerge model while the original GoogLeNet can only allow a batch size of 350. On the other hand, SqueezeNet though has much less trained parameters, it has much larger run-time memory impact due to the increased number of layers.
4.2 ALEXNET AND RESNET
To further analyze the generality of proposed DeepRebirth acceleration framework, besides GoogLeNet, we also apply the proposed framework to other popular deep neural structures: AlexNet (Krizhevsky et al.) and ResNet (He et al. (2015)). Note that we did not apply tensor weights compression to those two models which can further reduce the model forwarding latency.
First, we study the classical AlexNet model. We apply streamline merging approach to re-generate new layers by merging the first two convolution layers followed by LRN layers. We illustrate the result in Table 6. This indicates that by applying merging to the first two layers, the model forwarding time of AlexNet is reduced from 445 ms to 274 ms on Samsung Galaxy S5, and the Top-5 accuracy is slightly dropped from 80.03% to 79.57%.
We also apply the acceleration scheme to the state-of-the-art ResNet model. In the experiment, we use the popular 50-layer ResNet-50 model as baseline. We mainly apply the acceleration framework to conv1 and res2a layers (res2a has 2 branches; one branch has 1 convolution layer and another branch has 3 convolution layers). We present the result in Table 7. The time latency on Samsung Galaxy S5 for the processed layers (i.e., conv1 and res2a) is reduced from 189 ms to 104 ms. Moreover, the run-time memory cost is reduced by 2.21x. The accuracy is only slightly reduced.
5 RELATED WORK
Reducing the model size and accelerating the running speed are two general ways to facilitate the deployment of deep learning models on mobile devices. Many efforts have been spent on improving the model size. In particular, most works focus on optimizing tensor-layers to reduce the model size due to the high redundancy in the learned parameters in tensor layers of a given deep model. Vanhoucke et al. (2011) proposed a fixed-point implementation with 8-bit integer activation to reduce the number of parameter used in the deep neural network while Gong et al. (2014) applied vector quantization to compressed deep convnets. These approaches, however, mainly focus on compressing the fully connected layer without considering the convolutional layers. To reduce the parameter size, Denton et al. (2014) applied the low-rank approximation approach to compress the neural networks with linear structures. Afterwards, hashing function was utilized by Chen et al. (2015) to reduce model sizes by randomly grouping connection weights. More recently, Han et al. (2016b) proposed to effectively reduce model size and achieve speed-up by the combination of pruning, huffman coding and quantization. However, the benefits can only be achieved by running the compressed model on a specialized processor Han et al. (2016a). In general, reducing the model size can help deployment of deep learning models, this, however, does not necessarily bring significant speed up for running deep learning models. Compared to these works, instead of reducing model size, DeepRebirth provides a generic framework to accelerate the running speed that can be applied for different deep learning architectures on different low-level hardware (CPU, GPU, etc).
In order to improve the network running efficiency, some scalable networks have been proposed by balancing the running speed and the accuracy. Rastegari et al. (2016) designed a binary deep learning network (called XNOR-Net) where both the network weight and the input can be binarized for memory and computational saving. However, this network design depressed the accuracy greatly. The top-5 accuracy obtained by this framework is reduced by more than 10% for ResNet-18 model along with 2x speed-up. Another popular newly designed small model -SqueezeNet Iandola et al. (2016) becomes widely used for its much smaller memory cost and increased speed. However, the near-AlexNet accuracy is far below the state-of-the art performance. Compared with these two newly networks, our approach has much better accuracy with more significant acceleration.
Springenberg et al. (2014) shows that the conv-relu-pool substructure may not be necessary for a neural network architecture. The authors find that max-pooling can simply be replaced by another convolution layer with increased stride without loss in accuracy. Different from this work, DeepRebirth replaces a complete substructure (e.g., conv-relu-pool, conv-relu-LRN-pool) with a single convolution layer, and aims to speed-up the model execution on a mobile device. In addition, our work fine-tunes a trained network by relearning the merged “rebirth” layers and does not require to train from scratch.
6 CONCLUSION
We have proposed DeepRebirth acceleration framework which can speed up the neural networks with satisfactory accuracy. Our method operates by re-generating new tensor layers from optimizing the non-tensor layers and their neighboring units. Moreover, as a generic method, DeepRebirth is compatible with state-of-the-art deep models like GoogleNet and ResNet, where most parameter weight compression methods failed. By applying DeepRebirth at different deep learning architectures, we obtain the significant speed-up on different processors, especially on mobile CPUs. This will greatly facilitate the deployment of deep learning models on mobile phones and make it possible to provide more smart and intelligent services in the new AI tide.

This paper looks at the idea of fusing multiple layers (typically a convolution and a LRN or pooling layer) into a single convolution via retraining of just that layer, and shows that simpler, faster models can be constructed that way at minimal loss in accuracy. This idea is fine. Several issues:
- The paper introduces the concept of a 'Deeprebirth layer', and for a while it seems like it's going to be some new architecture. Mid-way, we discover that 1) it's just a convolution 2) it's actually a different kind of convolution depending on whether one fuses serial or parallel pooling layers. I understand the desire to give a name to the technique, but in this case naming the layer itself, when it's actually multiple things, non of which are new architecturally, confuses the argument a lot.
- There are ways to perform this kind of operator fusion without retraining, and some deep learning framework such as Theano and the upcoming TensorFlow XLA implement them. It would have been nice to have a baseline that implements it, especially since most of the additional energy cost from non-fused operators comes from the extra intermediate memory writes that operator fusion removes.
- Batchnorm can be folded into convolution layers without retraining by scaling the weights. Were they folded into the baseline figures reported in Table 7?
- At the time of writing, the authors have not provided the details that would make this research reproducible, in particular how the depth of the fused layers relates to the depth of the original layers in each of the experiments.
- Retraining: how much time (epochs) does the retraining take? Did you consider using any form of distillation?
Interesting set of experiments. This paper needs a lot of improvements to be suitable for publication.
- Open-sourcing: having the implementation be open-source always enhances the usefulness of such paper. Not a requirement obviously.


The Neural GPU is a recent model that can learn algorithms such as multi-digit binary addition and binary multiplication in a way that generalizes to inputs of arbitrary length. We show that there are two simple ways of improving the performance of the Neural GPU: by carefully designing a curriculum, and by increasing model size. The latter requires a memory efficient implementation, as a naive implementation of the Neural GPU is memory intensive. We find that these techniques increase the set of algorithmic problems that can be solved by the Neural GPU: we have been able to learn to perform all the arithmetic operations (and generalize to arbitrarily long numbers) when the arguments are given in the decimal representation (which, surprisingly, has not been possible before). We have also been able to train the Neural GPU to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of the operands, although these have succeeded only in their binary representation, and not with perfect accuracy. In addition, we gain insight into the Neural GPU by investigating its failure modes. We find that Neural GPUs that correctly generalize to arbitrarily long numbers still fail to compute the correct answer on highly-symmetric, atypical inputs: for example, a Neural GPU that achieves near-perfect generalization on decimal multiplication of up to 100-digit long numbers can fail on 000000 . . . 002× 000000 . . . 002 while succeeding at 2 × 2. These failure modes are reminiscent of adversarial examples.
1 INTRODUCTION
Deep Neural Networks (DNNs) are extremely useful for solving difficult pattern recognition tasks for two reasons: first, because they can compactly represent good solutions to difficult pattern recognition tasks; and second, because these good solutions can be found with stochastic gradient descent. It is not immediately obvious that a DNN should have the ability to represent solutions to such problems compactly. This is the case because the depth and width of DNNs allows them to simulate any parallel computer that runs for a modest number of steps, making it possible for the DNN to match the performance of any parallelizeable statistical ML model by simply simulating it. This is one of the reasons DNNs are successful relative to other ML models.
DNNs are especially useful in the supervised learning setting, where the goal is achieve low test error over a given data distribution. Statistical learning theory (Vapnik, 2013) guarantees that this can be done by minimizing training error, as long as the training data is drawn from the same data distribution and when there is more training cases than parameters.
While the ability to achieve low test error on a specific data distribution is extremely useful in practice and has already enabled a wide range of practical applications, there is evidence that this ability does not fully capture the intuitive notion of pattern recognition. For example, the existence of adversarial examples, which are data cases that are nearly indistinguishable from real data cases that confuse all existing discriminative classifiers, suggests that supervised DNNs are substantially less robust than human pattern recognition. And indeed, we would expect a system that has fully “understood” the relevant visual (say) concepts to not be fooled by adversarial examples. Understanding and fixing this problem is an active area of research.
1Work done while at OpenAI.
Another domain where a mere low error on a specific data distribution seems unsatisfying is the domain of simple algorithms. Simple algorithms have a well-defined output for all conceivable inputs, so if we collect enough of input-output examples from a reasonably-chosen distribution where the outputs are computed by some (unknown) simple algorithm, a sufficiently good learning method ought to be able to infer the “true” algorithm: one that can perfectly generalize to all conceivable inputs, and not just the inputs that tend to occur in the training data distribution.
This problem lies at the core of program induction, an old field that has significant past work (Nordin, 1997; Liang et al., 2013; Wineberg & Oppacher, 1994; Solomonoff, 1964; Holland, 1992; Goldberg, 1989; Gomez et al., 2008). More recently, researchers have begun investigating this problem using the deep learning techniques of neural network function approximation and stochastic gradient descent (Graves et al., 2014; Zaremba & Sutskever, 2015; Kaiser & Sutskever, 2015; Kurach et al., 2015; Andrychowicz & Kurach, 2016). All these works have been able to learn algorithms that correctly generalize to inputs of much greater length than the training cases on some problems. The Neural GPU (Kaiser & Sutskever, 2015) is notable among these since it is the only model that has, thus far, been able to learn to correctly multiply integers of length much greater than it has been trained on.
The phenomenon of generalization to inputs that are outside the training distribution is poorly understood. The space of problems for which such generalization is possible has not been identified, and a detailed understanding of the causes of such generalization are missing as well. Given that the test inputs do not come from the training data distribution, we do not yet have a formal reason to believe that such out-of-distribution generalization should succeed.
In this paper, we attempt to better understand this generalization in the context of the Neural GPU. We empirically study the parameters that affect its probability of successfully generalizing to inputs much greater length, and also study its failures. We report three notable findings: first, that larger models can generalize on harder tasks; second, that very detailed curriculum can enable the training of otherwise un-trainable neural networks; and third, those models that achieve perfect generalization on longer test cases when they are drawn from the uniform distribution still fail on highly structured inputs. This suggests that these models fail to learn the “true algorithm” as well as we’ve hoped, and that additional research is required for to learn models that can generalize much better.
The code for this work can be found at https://github.com/openai/ ecprice-neural-gpu.
2 RELATED WORK
The problem of learning algorithms from data has been investigated in the field of program synthesis (Nordin, 1997; Liang et al., 2013; Wineberg & Oppacher, 1994; Solomonoff, 1964) and genetic programming (Holland, 1992; Goldberg, 1989; Gomez et al., 2008). These approaches usually aim to directly produce the source code of an algorithm that solves the problem specified by the training data.
A recent approach to algorithm learning attempts to use the power of neural networks and their learning algorithm. Neural networks are inherently robust, and are capable of dealing with “imprecise” data (such as images or text) that can be difficult to handle in models that directly work with source code. There exist a number of neural network architecture that implement this idea: the Neural Turing Machine (NTM) (Graves et al., 2014), the standard LSTM (to some extent) (Zaremba & Sutskever, 2014), the Grid LSTM (Kalchbrenner et al., 2015), the Stack RNN (Joulin & Mikolov, 2015), the Neural DeQue (Grefenstette et al., 2015), the End-to-End Memory Networks (Sukhbaatar et al., 2015), the Hierarchical Attentive Memory (Andrychowicz & Kurach, 2016), and the Neural random-access machines (Kurach et al., 2015).
For a neural model to be able to learn an algorithm, it is essential that it is capable of running the necessary number of computational steps. Most of the above models have only been successfully used to learn algorithms that require a linear number of computational steps in the size of the input. While some models (Zaremba & Sutskever, 2015; Zaremba et al., 2015; Graves, 2016) can in principle learn the correct runtime for a given algorithm, in practice it has not been possible to learn algorithms requiring superlinear runtime, such as long integer multiplication. The only known
neural model that can solve tasks whose runtime is truly superlinear in the size of the input is the Neural GPU (Kaiser & Sutskever, 2015), which is the model that we investigate further in this paper.
The Grid LSTM (Kalchbrenner et al., 2015) is a powerful architecture that has been used to successfully learn 15-digit decimal addition. This model is similar to the Neural GPU – the main difference is that the Neural GPU is less recurrent. The Neural GPU has been shown to be able to generalize outside its training distribution, and while this has not been shown for the Grid LSTM, we believe that with appropriate modification, it should be capable of such generalization as well.
Neural network models that are capable of learning and representing algorithms tend to be extremely deep and have an elaborate architecture, which makes the problem of minimizing their training error to be very challenging for stochastic gradient descent. To reduce the difficulty of the training problem, curriculum learning has been found to be critical. In particular, all the aforementioned models require a curriculum learning (Bengio et al., 2009; Zaremba & Sutskever, 2014) in order to successfully learn sophisticated functions, and the results in this paper are no different.
3 MODEL
The Neural GPU is an architecture developed by Kaiser & Sutskever (2015) that can learn algorithms such as binary multi-digit addition and multi-digit multiplication. The main idea of the Neural GPU is to build an architecture capable of performing the necessary computation for the task without being overly deep. By being less deep, the problem of minimizing its training error becomes easier. Tasks that require a super-linear number of computational operations (in the size of the input) cannot be solved by a neural architecture that can only perform a linear number computational operation. Table 1 lists the number of computational operations that can be performed by several different neural network architectures. In our notation, a feed forward network consumes input of a predefined, constant size O ( 1 ) , and performs a fixed number of computation operations (also O ( 1 ) ). Other
architectures, such as classical convolution networks, also have a predefined input size (O ( 1 ) ), and
process it with a constant O ( 1 )
number of computational operations. However, it is possible to apply the convolution operation to an input of variable size. This approach is sometimes used in object detection, where the same convolutional neural network is applied on images of variable size (Sermanet et al., 2013). Similarly, the recurrent neural network (RNN) is capable of processing inputs of variable length, where the amount of computation performed by the RNN is linear in the length of the sequence.
The Neural GPU architecture is the combination of a convolution on variable size inputs with a recurrent neural network as shown in Fig. 1. The Neural GPU consumes an input of a variable length n. It repeatedly applies several convolutional layers n times, where n is the length of the input; thus, the depth of the model is dependent on the length of the input. It performs O ( n2 )
operations for each input of the length O ( n ) . This is a desirable property, because now there is a possibility that the model could learn to represent algorithms whose runtime grows superlinearly in the size of the input, such as integer multiplication. Harder problems can require significantly more computational steps. While we could, in principle, unroll our models for an enormous number of
timesteps to make them capable of solving even NP-hard problems, it is exceedingly unlikely that gradient learning would succeed training models of such depth.
The architecture most similar to Neural GPU is the Grid LSTM (Kalchbrenner et al., 2015). It has been shown to learn 15 digit long decimal addition task, although it has not yet been shown to generalize to inputs of length greater than the training data.
To successfully train the Neural GPU, Kaiser & Sutskever (2015) used the following techniques:
• The architecture is that of a gated recurrent unit (GRU) through depth (Bahdanau et al., 2014).
• Tanh cutoff: the hyperbolic tangent activations used by the GRU are truncated once they reach a critical upper (and lower) bound. The hope is that this makes the results more “digital”.
• The use of Dropout (Srivastava et al., 2014). • Instead of using the same weights at every pair of layers, the Neural GPU starts out by
cycling through 6 independent sets of weights, which are gradually annealed to become identical as optimization progresses.
• Gradient noise: the original paper notes it as important, but the released code has this option disabled.
In this paper, we use all these techniques and analyze their importance Fig. 2. More details about the meaning of each modification are available in the original work (Kaiser & Sutskever, 2015).
4 IMPROVEMENTS TO THE NEURAL GPU
The performance obtained by the Neural GPU is very sensitive to the random seed used to produce the initialization and the order of the training data. For example, when we ran several identical models with different random seeds, we find, similarly to Kaiser & Sutskever (2015), that only a fraction of them generalizes to over 99% cases (where the test cases are of length much greater than the training cases, and the model is considered to have made an error when it mispredicts even one bit of the output vector). Such variability makes it hard to hill-climb in hyperparameter space. We address this problem by running ≈ 15 instances of the each model with different seeds on each of our 13 tasks. We measure the fraction of seeds that cause the model to surpass 99% accuracy on the (longer and out of distribution) test data. This fraction provides us with a reliable estimate of the quality of the hyperparameter configuration. Moreover, such measurements had to be done on several problems. Particular “tricks” often are effective just for a single problem, and damage performance on others.
Fig. 2 shows the success rate for a given size and model modification. We find that many model modifications help reduce training error but do not with generalization. Note that most changes to the model increase the success rate by no more than 10%, which is too small to be detected without the thorough experimental framework that we’ve outlined above.
Our experiments in Fig. 2 show that the simplest way to increase performance is to increase the size of the model. The rows of Table 2 consistently show that larger models are more likely to generalize. In contrast, smaller models are often unable to minimize training error. For example, we were not able to achieve 0% training error when models with 128 and 256 filters are trained on decimal multiplication task (left Fig. 3), while the larger models with 512 filters achieve 0% training
error (although they do not generalize to long test inputs since curriculum was not used). It is not self-evident that larger model would generalize better, but the empirical results are clear. We suspect that the over-parameterization of the model makes the optimization task easier.
0 50k 100k 150k 200k 0%
20%
40%
60%
80%
100%
Tr ai
ni ng
er ro
r
0 50k 100k 150k 200k Steps of training
0%
20%
40%
60%
80%
100%
Te st
er ro
r
Decimal multiplication is hard to train end-to-end
128 filters 256 filters
512 filters
0 50k 100k 150k 200k 0%
20%
40%
60%
80%
100%
Tr ai
ni ng
er ro
r
256 filters
0 50k 100k 150k 200k Steps of training
0%
20%
40%
60%
80%
100%
Te st
er ro
r
Decimal multiplication is easier with curriculum
Base 10 Base 2 to 10
Base 2 to 5 to 10 Base 2 to 4 to 10
Size and curriculum both help
Running models with 512 filters is challenging, because they do not fit into the memory of the GPU on the problems that we have been exploring. In the implementation used for the experiments in this paper, a Neural GPU with 128 filters does not fit into GPU memory (12 GB) due to its large number
of activations. To train larger models, we used several techniques for reducing the model’s memory footprint. First, instead of unrolling the graph, we used TensorFlow’s symbolic tf.while loop, which can store intermediate activations on CPU thanks to the swap memory option (implemented in tensorflow (Abadi, 2015)). Second, we have decreased the batch size, which further lowered the memory footprint. A third way would be to use the methods described by Martens & Sutskever (2012); Gruslys et al. (2016), however we have not experimented with them.
In its current form, the Neural GPU cannot reliably learn to solve hard tasks such as decimal multiplication from examples (although sporadically it does). We find that an elaborate curriculum is necessary in order to learn such problems reliably, and to solve harder tasks. Our curriculum involves starting with short examples first, and solving intermediate task on the way to solve the target task. For instance, a curriculum for decimal multiplication could consist of first learning multiplication in base 2, then base 4, and finally base 10. The second plot in Fig. 3 shows the training and test error for models trained with curriculum. The third plot in Fig. 3 presents results for different curriculum types and model sizes. Table 2 summarizes the success rates for different model sizes, and curriculum.
We have obtained other notable results on the 3-numbers multiplication task (Fig. 4). We were not able to reach perfect error when training on such this task without a curriculum. We found that training on multiplication of 2 numbers and then moving to multiplication of 3 numbers improved performance. However, such model doesn’t generalize. The best performance on the 3-numbers multiplication is achieved when each training instance consists l numbers, where l is randomly chosen between 1 and bn−12 c, where n is the length of a single number. Another experiment is to train a model on sequences of arithmetic operations with multiple numbers. We train on expressions of length 41, and test on expressions of length 201. When using the binary representation, our model is able to correctly solve 30% of length-201 expressions, where as always we count success only when the whole number is predicted correctly. For an example of successful generalization, the model successfully evaluated the expression 001110111001/1+10-10*0/1/1*111/10*010+0/010011-10/10101*0+ 010/1*00-110*1*0/1/101000- 00000+ 000-1+ 1-1011111*010-010/0011111011-010-1100-0/0010000*01*0010000+ 0111110+ 00001+10/10*111111111-10*10-1*11111+01 (whose value is 10100001011). Fig. 5 summarizes this result.
5 GENERALIZATION
Integer addition is a well defined algorithm, so knowledge of its operation is sufficient to add arbitrary numbers. Our trained model generalize perfectly to > 99% of uniformly random test cases of much greater length (having a single digit error in the whole 100-digit number is considered an error). However, we found that it fails on a much larger fraction of “structured” examples. For example, it fails on cases that involve carrying a digit for many steps.
The probability that a bit is to be carried k times, on uniformly random binary inputs, is 2−k. Training and testing on random examples will thus not notice failure to carry more than about 20 bits. We find that the model typically fails to generalize on test cases that require more than 20 steps of carry, as shown in Fig. 6. Similarly, with decimal addition, the chance of carrying k digits is 10−k and most trained models stop working reliably after 6 digits.
Introducing “hard” training instances with long carries does not solve this issue. Our training instances involve 20-digit numbers, so they can only carry for 20 digits. Indeed, we find that the trained model will usually answer length-20 carry inputs correctly, but not length-30.
The above behavior holds for most trained models, but as in Kaiser & Sutskever (2015) we find that training the Neural GPU with different random seeds will occasionally find models that generalize better. We trained a decimal addition model with 1121 different seeds, and measure for each model the number of carries at which the error rate crosses 50%. As shown on the right of Fig. 6, the distribution is bimodal: 94% of models have a threshold between 7 and 9 digits, while almost the entire remainder (5.5%) have a threshold of at least length 33. These “successful” models seem to follow roughly a power law distribution: 5% are at least 33, 1% are at least 52, and our best performing model worked up to a carry of length 112. If this behavior is indeed a power law, it is not a very promising way to find highly-generalizing models: a rough extrapolation suggests generalizing to 1000 digits would require 200 million trials.
We observe similar generalization issues with multiplication. A model that has < 1% error on two random 100-digit long decimal examples fails much more often on very symmetric numbers and on numbers that requires to carry for many steps (we test it by verifying performance on numbers that multiply to b100 or b100 − 1). It also failed much more often on multiplying single digit numbers when they are prepended with many zeros (Table 3); one random trained model failed on 38 of the 100 possible pairs of single-digit numbers.
The multiplication model also fails on highly regular cases. The outputs of the multiplication model are similar to the correct answer, however it seems that the model made a mistake in the middle of the computation. For instance, we took two numbers that multiply to 10180 − 1. Such a number is full of nines. However, model predicted: 10180 + 10103 − 1 instead. The predicted number differs on 78 positions, because it has many zeros instead of nines; at some point, it starts to guess that it is getting a carry. Moreover, the model has trouble with very structured expressions, such as 1111111111× 1111111111. It fails for any number of ones above 8.
6 GLOBAL OPERATION
The Neural GPU is a cellular automaton, which is a Turing complete computational model (Chopard & Droz, 1998; Wolfram, 1984). However, the automaton is often computationally inefficient compared to the von Neumann architecture. It is difficult for a cellular automaton to move data globally as the entirety of its computation operates locally at each step. We wanted to understand the importance of globally moving data around for solving algorithmic tasks.
Alignment helps addition, hurts multiplication
In principle, the Neural GPU could be made more powerful by adding a global operation in each of its layers. We have briefly tried to include an attention mechanism that has the ability to easily shift rectangles of data to locations specified by the model, but we were not able to improve empirical results. We skip these experiments here, and instead investigate a simpler form of this question.
Given that the Neural GPU cannot easily move information across long distances, it is conceivable that having both arguments concatenated would hurt its performance. We therefore experimented with several input representations, as described below:
• Padded: 12345+00067 • Unpadded: 12345+67
• Aligned: 12345+00067
Could we manually align data in a way that helps to solve a task? This could indicate that an architectural modification that performs a global operation is needed. And indeed, we found that the addition task on aligned numbers has higher success rate than on concatenated numbers, and addition on unpadded numbers is the hardest one (Fig. 7, left).
However, we observe the opposite outcome for the multiplication task (Fig. 7, right). Empirically, aligning numbers for multiplication makes the task more difficult. These results suggest that an architectural modification that makes it easy to move the data globally need not provide an improvement on all problems.
7 CONCLUSION
In this paper, we investigated the generalization ability of the Neural GPU. We have discovered that larger Neural GPUs generalize better, and provided examples of several curriculums that made it possible for the Neural GPU to solve tasks that it was not able to solve before. Finally, we showed that its generalization is incomplete, as while it has successfully generalized to longer inputs, there still exist highly structured test cases that cause the model to fail.
It is desirable to develop learning methods that can learn algorithms that achieve perfect generalization. One way of moving forward is to investigate ways in which the model can benefit from additional sources of information that are not present in the task itself.
8 ACKNOWLEDGMENT
We wish to thank Rafal Jozefowicz for useful discussions and comments.

The authors investigate the neural GPU model introduced by Kaiser and Sutskever. In section 3 they claim its performance is due to the O(n^2) number of steps it can perform for each example. In the subsequent section they highlight the importance of curriculum training and empirically show that larger models generalize better. In section 5 they construct examples that reveal failure modes. In the last section they compare the performance given different input formats.
The paper is well written. It contains an exhaustive set of experiments which provide insight into the details of training the neural GPU model. It pushes the boundary of algorithms that can be learned further. On the other hand, the paper seems to lack a coherent message. It also fails to provide any insight into the how and why of the observations made (i.e. why curriculum training is essential and why certain failure modes exist).
The introduction contains several statements which should be qualified or explained further. As far as I am aware statistical learning theory does not guarantee that empirical risk minimization is consistent when the number of parameters is larger than the number of examples; the generalization performance depends on the VC dimension of the function space instead. Furthermore, the suggested link between adversarial examples and learning algorithms is tenuous, and references or a further explanation should be provided for the contentious statement that deep neural networks are able to match the performance of any parallel machine learning algorithm.
The authors argue that the neural GPU performs O(n^2) “steps” on each example, which allows it to learn algorithms with super-linear complexity such as multiplication. This analysis seems to overlook the parallel nature of the neural GPU architecture: Both addition and multiplication have O(log n) time complexity when parallelism is used (cf. a carry-lookahead adder and a Wallace tree respectively).
In section 4 the authors show that their larger models generalize better, which they argue is not self-evident. However, since both training and test error decrease it is likely that the smaller models are underfitting, in which case it is not counter-intuitive at all that a larger model would have better generalization error.
It is interesting to see that progressively decreasing the number of terms and increasing the radix of the number system works well as a learning curriculum, although it would be nice to have a stronger intuitive or theoretical justification for the latter.
The final section claims that neural GPUs are cellular automata. Further justification for this statement would be useful, since cellular automata are discrete models, and the equivalence between both models is in no way obvious. The relationship between global operations and changing the input format is circuitous.
In conclusion, the paper provides some useful insights into the neural GPU model, but does not introduce original extensions to the model and does not explain any fundamental limitations. Several statements require stronger substantiation.
Pro:
* Well written
* Exhaustive set of experiments
* Learning algorithms with decimal representation
* Available source code
Cons:
* No coherent hypothesis/premise advanced
* Two or three bold statements without explanation or references
* Some unclarity in experimental details
* Limited novelty and originality factor
Typos: add minus in “chance of carrying k digits is 10^k” (section 5); remove “are” from “the larger models with 512 filters are achieve” (section 4); add “a” in “such model doesn’t generalize” (section 4).


RECTIFIED FACTOR NETWORKS FOR BICLUSTERING
Biclustering is evolving into one of the major tools for analyzing large datasets given as matrix of samples times features. Biclustering has several noteworthy applications and has been successfully applied in life sciences and e-commerce for drug design and recommender systems, respectively. FABIA is one of the most successful biclustering methods and is used by companies like Bayer, Janssen, or Zalando. FABIA is a generative model that represents each bicluster by two sparse membership vectors: one for the samples and one for the features. However, FABIA is restricted to about 20 code units because of the high computational complexity of computing the posterior. Furthermore, code units are sometimes insufficiently decorrelated. Sample membership is difficult to determine because vectors do not have exact zero entries and can have both large positive and large negative values. We propose to use the recently introduced unsupervised Deep Learning approach Rectified Factor Networks (RFNs) to overcome the drawbacks of existing biclustering methods. RFNs efficiently construct very sparse, non-linear, highdimensional representations of the input via their posterior means. RFN learning is a generalized alternating minimization algorithm based on the posterior regularization method which enforces non-negative and normalized posterior means. Each code unit represents a bicluster, where samples for which the code unit is active belong to the bicluster and features that have activating weights to the code unit belong to the bicluster. On 400 benchmark datasets with artificially implanted biclusters, RFN significantly outperformed 13 other biclustering competitors including FABIA. In biclustering experiments on three gene expression datasets with known clusters that were determined by separate measurements, RFN biclustering was two times significantly better than the other 13 methods and once on second place. On data of the 1000 Genomes Project, RFN could identify DNA segments which indicate, that interbreeding with other hominins starting already before ancestors of modern humans left Africa.
1 INTRODUCTION
Biclustering is widely-used in statistics (A. Kasim & Talloen, 2016), and recently it also became popular in the machine learning community (O´ Connor & Feizi, 2014; Lee et al., 2015; Kolar et al., 2011), e.g., for analyzing large dyadic data given in matrix form, where one dimension are the samples and the other the features. A matrix entry is a feature value for the according sample. A bicluster is a pair of a sample set and a feature set for which the samples are similar to each other on the features and vice versa. Biclustering simultaneously clusters rows and columns of a matrix. In particular, it clusters row elements that are similar to each other on a subset of column elements. In contrast to standard clustering, the samples of a bicluster are only similar to each other on a subset of features. Furthermore, a sample may belong to different biclusters or to no bicluster at all. Thus, biclusters can overlap in both dimensions. For example, in drug design biclusters are compounds which activate the same gene module and thereby indicate a side effect. In this example different chemical compounds are added to a cell line and the gene expression is measured (Verbist et al., 2015). If multiple pathways are active in a sample, it belongs to different biclusters and may
have different side effects. In e-commerce often matrices of costumers times products are available, where an entry indicates whether a customer bought the product or not. Biclusters are costumers which buy the same subset of products. In a collaboration with the internet retailer Zalando the biclusters revealed outfits which were created by customers which selected certain clothes for a particular outfit.
FABIA (factor analysis for bicluster acquisition, (Hochreiter et al., 2010)) evolved into one of the most successful biclustering methods. A detailed comparison has shown FABIA’s superiority over existing biclustering methods both on simulated data and real-world gene expression data (Hochreiter et al., 2010). In particular FABIA outperformed non-negative matrix factorization with sparseness constraints and state-of-the-art biclustering methods. It has been applied to genomics, where it identified in gene expression data task-relevant biological modules (Xiong et al., 2014). In the large drug design project QSTAR, FABIA was used to extract biclusters from a data matrix that contains bioactivity measurements across compounds (Verbist et al., 2015). Due to its successes, FABIA has become part of the standard microarray data processing pipeline at the pharmaceutical company Janssen Pharmaceuticals. FABIA has been applied to genetics, where it has been used to identify DNA regions that are identical by descent in different individuals. These individuals inherited an IBD region from a common ancestor (Hochreiter, 2013; Povysil & Hochreiter, 2014). FABIA is a generative model that enforces sparse codes (Hochreiter et al., 2010) and, thereby, detects biclusters. Sparseness of code units and parameters is essential for FABIA to find biclusters, since only few samples and few features belong to a bicluster. Each FABIA bicluster is represented by two membership vectors: one for the samples and one for the features. These membership vectors are both sparse since only few samples and only few features belong to the bicluster.
However, FABIA has shortcomings, too. A disadvantage of FABIA is that it is only feasible with about 20 code units (the biclusters) because of the high computational complexity which depends cubically on the number of biclusters, i.e. the code units. If less code units were used, only the large and common input structures would be detected, thereby, occluding the small and rare ones. Another shortcoming of FABIA is that units are insufficiently decorrelated and, therefore, multiple units may encode the same event or part of it. A third shortcoming of FABIA is that the membership vectors do not have exact zero entries, that is the membership is continuous and a threshold have to be determined. This threshold is difficult to adjust. A forth shortcoming is that biclusters can have large positive but also large negative members of samples (that is positive or negative code values). In this case it is not clear whether the positive pattern or the negative pattern has been recognized.
Rectified Factor Networks (RFNs; (Clevert et al., 2015)) RFNs overcome the shortcomings of FABIA. The first shortcoming of only few code units is avoided by extending FABIA to thousands of code units. RFNs introduce rectified units to FABIA’s posterior distribution and, thereby, allow for fast computations on GPUs. They are the first methods which apply rectification to the posterior distribution of factor analysis and matrix factorization, though rectification it is well established in Deep Learning by rectified linear units (ReLUs). RFNs transfer the methods for rectification from the neural network field to latent variable models. Addressing the second shortcoming of FABIA, RFNs achieve decorrelation by increasing the sparsity of the code units using dropout from field of Deep Learning. RFNs also address the third FABIA shortcoming, since the rectified posterior means yield exact zero values. Therefore, memberships to biclusters are readily obtained by values that are not zero. Since RFNs only have non-negative code units, the problem of separating the negative from the positive pattern disappears.
2 IDENTIFYING BICLUSTERS BY RECTIFIED FACTOR NETWORKS
2.1 RECTIFIED FACTOR NETWORKS
We propose to use the recently introduced Rectified Factor Networks (RFNs; (Clevert et al., 2015)) for biclustering to overcome the drawbacks of the FABIA model. The factor analysis model and the construction of a bicluster matrix are depicted in Fig. 1. RFNs efficiently construct very sparse, nonlinear, high-dimensional representations of the input. RFN models identify rare and small events in the input, have a low interference between code units, have a small reconstruction error, and explain the data covariance structure.
RFN learning is a generalized alternating minimization algorithm derived from the posterior regularization method which enforces non-negative and normalized posterior means. These posterior means are the code of the input data. The RFN code can be computed very efficiently. For nonGaussian priors, the computation of the posterior mean of a new input requires either to numerically solve an integral or to iteratively update variational parameters. In contrast, for Gaussian priors the posterior mean is the product between the input and a matrix that is independent of the input. RFNs use a rectified Gaussian posterior, therefore, they have the speed of Gaussian posteriors but lead to sparse codes via rectification. RFNs are implemented on GPUs.
The RFN model is a factor analysis model
v = Wh + ✏ , (1)
which extracts the covariance structure of the data. The prior h ⇠ N (0, I) of the hidden units (factors) h 2 Rl and the noise ✏ ⇠ N (0, ) of visible units (observations) v 2 Rm are independent. The model parameters are the weight (factor loading) matrix W 2 Rm⇥l and the noise covariance matrix 2 Rm⇥m. RFN models are selected via the posterior regularization method (Ganchev et al., 2010). For data {v} = {v1, . . . ,vn}, it maximizes the objective F :
F = 1 n
nX
i=1
log p(vi) 1
n
nX
i=1
DKL(Q(hi | vi) k p(hi | vi)), (2)
where DKL is the Kullback-Leibler distance. Maximizing F achieves two goals simultaneously: (1) extracting desired structures and information from the data as imposed by the generative model and (2) ensuring sparse codes via Q from the set of rectified Gaussians.
For Gaussian posterior distributions, and mean-centered data {v} = {v1, . . . ,vn}, the posterior p(hi | vi) is Gaussian with mean vector (µp)i and covariance matrix ⌃p:
(µp)i = I + W T 1W 1 W T 1 vi , ⌃p = I + W T 1W 1 . (3)
For rectified Gaussian posterior distributions, ⌃p remains as in the Gaussian case, but minimizing the second DKL of Eq. (2) leads to constrained optimization problem (see Clevert et al. (2015))
min
µi
1
n
nX
i=1
(µi (µp)i)T ⌃ 1p (µi (µp)i)
s.t. 8i : µi 0 , 8j : 1
n
nX
i=1
µ2ij = 1 , (4)
where “ ” is component-wise. In the E-step of the generalized alternating minimization algorithm (Ganchev et al., 2010), which is used for RFN model selection, we only perform a step of the gradient projection algorithm (Bertsekas, 1976; Kelley, 1999), in particular a step of the projected Newton method for solving Eq. (4) (Clevert et al., 2015). Therefore, RFN model selection is extremely efficient but still guarantees the correct solution.
2.2 RFN BICLUSTERING
For a RFN model, each code unit represents a bicluster, where samples, for which the code unit is active, belong to the bicluster. On the other hand features that activates the code unit belong to the bicluster, too. The vector of activations of a unit across all samples is the sample membership vector. The weight vector which activates the unit is the feature membership vector. The un-constraint posterior mean vector is computed by multiplying the input with a matrix according to Eq. (3). The constraint posterior of a code unit is obtained by multiplying the input by a vector and subsequently rectifying and normalizing the code unit (Clevert et al., 2015).
To keep feature membership vector sparse, we introduce a Laplace prior on the parameters. Therefore only few features contribute to activating a code unit, that is, only few features belong to a bicluster. Sparse weights Wi are achieved by a component-wise independent Laplace prior for the weights:
p(Wi) = ⇣
1p 2
⌘n nY
k=1
e p 2 |Wki| (5)
The weight update for RFN (Laplace prior on the weights) is
W = W + ⌘ U S 1 W
↵ sign(W ) . (6)
Whereby the sparseness of the weight matrix can be controlled by the hyper-parameter ↵ and U and S are defined as U = 1n Pn i=1 viµ T i and S = 1 n Pn i=1 µiµ T i +⌃, respectively. In order to enforce more sparseness of the sample membership vectors, we introduce dropout of code units. Dropout means that during training some code units are set to zero at the same time as they get rectified. Dropout avoids co-adaptation of code units and reduces correlation of code units — a problem of FABIA which is solved.
RFN biclustering does not require a threshold for determining sample memberships to a bicluster since rectification sets code units to zero. Further crosstalk between biclusters via mixing up negative and positive memberships is avoided, therefore spurious biclusters do less often appear.
3 EXPERIMENTS
In this section, we will present numerical results on multiple synthetic and real data sets to verify the performance of our RFN biclustering algorithm, and compare it with various other biclustering methods.
3.1 METHODS COMPARED
To assess the performance of rectified factor networks (RFNs) as unsupervised biclustering methods, we compare the following 14 biclustering methods:
(1) RFN: rectified factor networks (Clevert et al., 2015), (2) FABIA: factor analysis with Laplace prior on the hidden units (Hochreiter et al., 2010; Hochreiter, 2013), (3) FABIAS: factor analysis with sparseness projection (Hochreiter et al., 2010), (4) MFSC: matrix factorization with sparseness constraints (Hoyer, 2004), (5) plaid: plaid model (Lazzeroni & Owen, 2002; T. Chekouo & Raffelsberger, 2015), (6) ISA: iterative signature algorithm (Ihmels et al., 2004), (7) OPSM: orderpreserving sub-matrices (Ben-Dor et al., 2003), (8) SAMBA: statistical-algorithmic method for bicluster analysis (Tanay et al., 2002), (9) xMOTIF: conserved motifs (Murali & Kasif, 2003), (10) Bimax: divide-and-conquer algorithm (Prelic et al., 2006), (11) CC: Cheng-Church -biclusters
(Cheng & Church, 2000), (12) plaid t: improved plaid model (Turner et al., 2003), (13) FLOC: flexible overlapped biclustering, a generalization of CC (Yang et al., 2005), and (14) spec: spectral biclustering (Kluger et al., 2003).
For a fair comparison, the parameters of the methods were optimized on auxiliary toy data sets. If more than one setting was close to the optimum, all near optimal parameter settings were tested. In the following, these variants are denoted as method variant (e.g. plaid ss). For RFN we used the following parameter setting: 13 hidden units, a dropout rate of 0.1, 500 iterations with a learning rate of 0.1, and set the parameter ↵ (controlling the sparseness on the weights) to 0.01.
3.2 SIMULATED DATA SETS WITH KNOWN BICLUSTERS
In the following subsections, we describe the data generation process and results for synthetically generated data according to either a multiplicative or additive model structure.
3.2.1 DATA WITH MULTIPLICATIVE BICLUSTERS
We assumed n = 1000 genes and l = 100 samples and implanted p = 10 multiplicative biclusters. The bicluster datasets with p biclusters are generated by following model:
X = pX
i=1
i z T i + ⌥ , (7)
where ⌥ 2 Rn⇥l is additive noise; i 2 Rn and zi 2 Rl are the bicluster membership vectors for the i-th bicluster. The i’s are generated by (i) randomly choosing the number N i of genes in bicluster i from {10, . . . , 210}, (ii) choosing N i genes randomly from {1, . . . , 1000}, (iii) setting i components not in bicluster i to N (0, 0.22) random values, and (iv) setting i components that are in bicluster i to N (±3, 1) random values, where the sign is chosen randomly for each gene. The zi’s are generated by (i) randomly choosing the number Nzi of samples in bicluster i from {5, . . . , 25}, (ii) choosing Nzi samples randomly from {1, . . . , 100}, (iii) setting zi components not in bicluster i to N (0, 0.22) random values, and (iv) setting zi components that are in bicluster i to N (2, 1) random values. Finally, we draw the ⌥ entries (additive noise on all entries) according to N (0, 32) and compute the data X according to Eq. (7). Using these settings, noisy biclusters of random sizes between 10⇥5 and 210⇥25 (genes⇥samples) are generated. In all experiments, rows (genes) were standardized to mean 0 and variance 1.
3.2.2 DATA WITH ADDITIVE BICLUSTERS
In this experiment we generated biclustering data where biclusters stem from an additive two-way ANOVA model:
X = pX
i=1
✓i ( i zTi ) + ⌥ , ✓ikj = µi + ↵ik + ij , (8)
where is the element-wise product of matrices and both i and zi are binary indicator vectors which indicate the rows and columns belonging to bicluster i. The i-th bicluster is described by an ANOVA model with mean µi, k-th row effect ↵ik (first factor of the ANOVA model), and jth column effect ij (second factor of the ANOVA model). The ANOVA model does not have interaction effects. While the ANOVA model is described for the whole data matrix, only the effects on rows and columns belonging to the bicluster are used in data generation. Noise and bicluster sizes are generated as in previous Subsection 3.2.1.
Data was generated for three different signal-to-noise ratios which are determined by distribution from which µi is chosen: A1 (low signal) N (0, 22), A2 (moderate signal) N (±2, 0.52), and A3 (high signal) N (±4, 0.52), where the sign of the mean is randomly chosen. The row effects ↵ki are chosen from N (0.5, 0.22) and the column effects ij are chosen from N (1, 0.52).
3.2.3 RESULTS ON SIMULATED DATA SETS
For method evaluation, we use the previously introduced biclustering consensus score for two sets of biclusters (Hochreiter et al., 2010), which is computed as follows:
Step (3) penalizes different numbers of biclusters in the sets. The highest consensus score is 1 and only obtained for identical sets of biclusters.
Table 1 shows the biclustering results for these data sets. RFN significantly outperformed all other methods (t-test and McNemar test of correct elements in biclusters).
3.3 GENE EXPRESSION DATA SETS
In this experiment, we test the biclustering methods on gene expression data sets, where the biclusters are gene modules. The genes that are in a particular gene module belong to the according bicluster and samples for which the gene module is activated belong to the bicluster. We consider three gene expression data sets which have been provided by the Broad Institute and were previously clustered by Hoshida et al. (2007) using additional data sets. Our goal was to study how well biclustering methods are able to recover these clusters without any additional information. (A) The “breast cancer” data set (van’t Veer et al., 2002) was aimed at a predictive gene signature for the outcome of a breast cancer therapy. We removed the outlier array S54 which leads to a data set with 97 samples and 1213 genes. In Hoshida et al. (2007), three biologically meaningful subclasses were found that should be re-identified. (B) The “multiple tissue types” data set (Su et al., 2002) are gene expression profiles from human cancer samples from diverse tissues and cell lines. The data set contains 102 samples with 5565 genes. Biclustering should be able to re-identify the tissue types. (C) The “diffuse large-B-cell lymphoma (DLBCL)” data set (Rosenwald et al., 2002) was aimed at predicting the survival after chemotherapy. It contains 180 samples and 661 genes. The three classes found by Hoshida et al. (2007) should be re-identified.
For methods assuming a fixed number of biclusters, we chose five biclusters — slightly higher than the number of known clusters to avoid biases towards prior knowledge about the number of actual clusters. Besides the number of hidden units (biclusters) we used the same parameters as described in Sec. 3.1. The performance was assessed by comparing known classes of samples in the data sets with the sample sets identified by biclustering using the consensus score defined in Subsection 3.2.3 — here the score is evaluated for sample clusters instead of biclusters. The biclustering results are summarized in Table 2. RFN biclustering yielded in two out of three datasets significantly better results than all other methods and was on second place for the third dataset (significantly according to a McNemar test of correct samples in clusters).
3.4 1000 GENOMES DATA SETS
In this experiment, we used RFN for detecting DNA segments that are identical by descent (IBD). A DNA segment is IBD in two or more individuals, if they have inherited it from a common ancestor, that is, the segment has the same ancestral origin in these individuals. Biclustering is well-suited to detect such IBD segments in a genotype matrix (Hochreiter, 2013; Povysil & Hochreiter, 2014), which has individuals as row elements and genomic structural variations (SNVs) as column elements. Entries in the genotype matrix usually count how often the minor allele of a particular SNV is present in a particular individual. Individuals that share an IBD segment are similar to each other because they also share minor alleles of SNVs (tagSNVs) within the IBD segment. Individuals that share an IBD segment represent a bicluster.
For our IBD-analysis we used the next generation sequencing data from the 1000 Genomes Phase 3. This data set consists of low-coverage whole genome sequences from 2,504 individuals of the main continental population groups (Africans (AFR), Asians (ASN), Europeans (EUR), and Admixed Americans (AMR)). Individuals that showed cryptic first degree relatedness to others were removed, so that the final data set consisted of 2,493 individuals. Furthermore, we also included archaic human and human ancestor genomes, in order to gain insights into the genetic relationships between humans, Neandertals and Denisovans. The common ancestor genome was reconstructed from human, chimpanzee, gorilla, orang-utan, macaque, and marmoset genomes. RFN IBD detec-
tion is based on low frequency and rare variants, therefore we removed common and private variants prior to the analysis. Afterwards, all chromosomes were divided into intervals of 10,000 variants with adjacent intervals overlapping by 5,000 variants
In the data of the 1000 Genomes Project, we found IBD-based indications of interbreeding between ancestors of humans and other ancient hominins within Africa (see Fig. 2 as an example of an IBD segment that matches the Neandertal genome).
4 CONCLUSION
We have introduced rectified factor networks (RFNs) for biclustering and benchmarked it with 13 other biclustering methods on artificial and real-world data sets.
On 400 benchmark data sets with artificially implanted biclusters, RFN significantly outperformed all other biclustering competitors including FABIA. On three gene expression data sets with previously verified ground-truth, RFN biclustering yielded twice significantly better results than all other methods and was once the second best performing method. On data of the 1000 Genomes Project, RFN could identify IBD segments which support the hypothesis that interbreeding between ancestors of humans and other ancient hominins already have taken place in Africa.
RFN biclustering is geared to large data sets, sparse coding, many coding units, and distinct membership assignment. Thereby RFN biclustering overcomes the shortcomings of FABIA and has the potential to become the new state of the art biclustering algorithm.
Acknowledgment. We thank the NVIDIA Corporation for supporting this research with several Titan X GPUs.

The paper presents a repurposing of rectified factor networks proposed
earlier by the same authors to biclustering. The method seems
potentially quite interesting but the paper has serious problems in
the presentation.
Quality:
The method relies mainly on techniques presented in a NIPS 2015 paper
by (mostly) the same authors. The experimental procedure should be
clarified further. The results (especially Table 2) seem to depend
critically upon the sparsity of the reported clusters, but the authors
do not explain in sufficient detail how the sparsity hyperparameter is
determined.
Clarity:
The style of writing is terrible and completely unacceptable as a
scientific publication. The text looks more like an industry white
paper or advertisement, not an objective scientific paper. A complete
rewrite would be needed before the paper can be considered for
publication. Specifically, all references to companies using your
methods must be deleted.
Additionally, Table 1 is essentially unreadable. I would recommend
using a figure or cleaning up the table by removing all engineering
notation and reporting numbers per 1000 so that e.g. "0.475 +/- 9e-4"
would become "475 +/- 0.9". In general figures would be preferred as a
primary means for presenting the results in text while tables can be
included as supplementary information.
Originality:
The novelty of the work appears limited: the method is mostly based on
a NIPS 2015 paper by the same authors. The experimental evaluation
appears at least partially novel, but for example the IBD detection is
very similar to Hochreiter (2013) but without any comparison.
Significance:
The authors' strongest claim is based on strong empirical performance
in their own benchmark problems. It is however unclear how useful this
would be to others as there is no code available and the details of
the implementation are less than complete. Furthermore, the method
depends on many specific tuning parameters whose tuning method is not
fully defined, leaving it unclear how to guarantee the generalisation
of the good performance.