We consider the problem of how to reduce the cost of communication that is required for the parallel training of a neural network. The state-of-the-art method, Bulk Synchronous Parallel Stochastic Gradient Descent (BSP-SGD), requires many collective communication operations, like broadcasts of parameters or reductions for partial gradient aggregations, which for large messages quickly dominates overall execution time and limits parallel scalability. To address this problem, we develop a new technique for collective operations, referred to as Linear Pipelining (LP). It is tuned to the message sizes that arise in BSP-SGD, and works effectively on multi-GPU systems. Theoretically, the cost of LP is invariant to P , where P is the number of GPUs, while the cost of the more conventional Minimum Spanning Tree (MST) scales like O(logP ). LP also demonstrates up to 2x higher bandwidth than Bidirectional Exchange (BE) techniques that are widely adopted by current MPI implementations. We apply these collectives to BSP-SGD, showing that the proposed implementations reduce communication bottlenecks in practice while preserving the attractive convergence properties of BSP-SGD.
1 INTRODUCTION
Scaling up neural networks with respect to parameter sizes, training sets, or both has drastically improved the state-of-the-art performance in several domains ranging from scene understanding, speech recognition, even to playing Go against professional players. Although training a large network saturated with nonlinearities is extremely time-consuming, the benefits brought forth by large-scale models has sparked a surge of interest in parallelizing training on multi-GPUs. The parallelization of SGD demands synchronizations to exchange gradients and parameters per iteration, and this introduces significant communication overhead. Previous studies have focused on trading the SGD convergence rate for fast gradient updates, such as stale or asynchronous SGD, 1-bit compressed gradient, etc. However, these methods are rarely adopted by Deep Learning frameworks as they depend on the balance between the enhanced iteration throughput and the decelerated convergence rate. Since BSP retains the convergence properties of SGD, its optimization should be of interest.
The gradient aggregations and parameter exchanges in BSP SGD are typical operations of communication collectives (Chan et al., 2007). Messages in the large-scale neural networks training are dense, long, and fixed-length, while the performance of collective algorithms is drastically sensitive to these attributes. Besides, the processing speed is several orders of magnitude faster than the network unidirectional transmission rate. These prioritize the utilization of network bandwidth in the collective design. However, we have seen sub-optimal collective algorithms, e.g. MST and BE, widely adopted by the deep learning community (Agarwal et al., 2014) (Jia et al., 2014) (Duchi et al., 2011). MST is only suitable for the latency dominant case such as frequent short message exchanges, while the bandwidth term of BE can be further improved (Thakur et al., 2005).
In this paper, we introduce new Linear Pipeline based collectives for multiGPU training. The collectives demonstrate O(log(P )) speedups over MST collectives and up to 2x speedups over BE based ones; the bounds only hold in training large neural networks. In particular, the theoretical analysis and the implementation yield an interesting insight that the cost of our design is invariant to GPU numbers, i.e., the cost of collective operations on 2 GPUs is similar to 20 GPUs. The design explores message granularity to maximize simultaneous bidirectional data exchanges. In specific, it divides a message into fine-grained blocks as the basic communication element. A GPU sends a block (via DMA 1) while receiving (via DMA 2) a new block from a neighbor. The copies are asynchronously launched on two GPU streams, and numerical operations further overlap data copies. As a result, our method yields a highly efficient pipeline over which messages for neural network training may be exchanged.
The proposed collective design achieves 2.3x to 360.55x speedups over Open MPI alternatives on 6 GPUs. In training GoogLeNet, we set up the same BSP SGD implementation with different underlying collectives. Our design demonstrates up to 1.7x convergence speedup over MST based Caffe.
2 RELATED WORK
The communication overhead has been widely identified as the major bottleneck in the data-parallel SGD (Shamir (2014), Li et al. (2014)). The data parallelism linearly adds the processing power by concurrent gradient computations with multiple GPUs. But it also requires synchronizations to collect partial gradients or to broadcast parameters. In practice, the communication rate is several orders of magnitude slower than the computation (Coates et al., 2013). Various approaches have been proposed to reduce the overhead.
The first group of approaches relaxes synchronous models of SGD to increase the iteration throughput (Dean et al. (2012), Zinkevich et al. (2010)). In this case, the relaxed SGD enables computations on a GPU to partially overlap with communications on others as demonstrated in Fig.1c and Fig.1d. Recht et al. (2011) proposed a lock free Asynchronous SGD (ASGD) that entirely gets rid of the synchronization requirement by allowing free concurrent parameter updates. But the relaxation only works well on sparse learning problems. In response, Ho et al. (2013) introduced the concept of staleness by bounding the fastest and the slowest machine within a few iterations of each other to ensure correctness. These relaxations claim to be effective as the enhanced iteration throughput offsets the disadvantages of degraded convergence rate. However, recent advances in deep learning frameworks (Cui et al. (2016)) have reestablished the advantages of BSP over relaxed ones in training neural networks. This reiterates the importance of studying BSP SGD.
The second group of approaches tries to reduce the overall communication volume. Seide et al. (2014) quantized gradients from 32 bits to 1 bit to reduce the message length, but the lost gradient information decelerates the convergence rate. Another approach is to accelerate the convergence with a large batch. Dekel et al. (2012) shows the convergence rate of mini-batch SGD isO(1/ √ Tb+1/T ) with b being the batch size. This result indicates a large batch needs fewer iterations to find a solution, and thereby fewer overall synchronizations. However, unwieldy increasing the batch size is also unfavorable under limited computing resources demonstrated by Wang et al. (2016b). Please note these methods still need synchronizations, and our work will further improve their performance.
The third group of approaches conducts system optimizations to minimize the communication cost (Wang et al., 2016a). Agarwal & Duchi (2011) and Agarwal et al. (2014) presented partial gradients aggregations guided with a MST that takes log(P ) steps to fully synchronize the model. Deep learning frameworks such as Caffe (Jia et al., 2014) also adopt this approach. Unfortunately, MST is only suitable for latency dominant scenarios (i.e. high frequent short messages). Although collective algorithms have been thoroughly discussed in the HPC community (Almási et al. (2005), Gabriel et al. (2004), Shipman et al. (2006)), few have studied their performances for the deep learning. The performance of collectives varies significantly with different message lengths and network topologies, while messages in deep network training are dense, long and fixed-length. Therefore, it is imperative to address such peculiarities in the collectives. Worringen (2003) proposed a pipeline collective model in shared memory environment for CPU data, but communications of different MPI processes sharing the same CPU memory bus within the same CPU socket. This causes bandwidth competition among different processes, thereby poor performance for the collective communication in shared memory environment for CPU data. In contrast, PCI-E is bi-directional. The latest GPUs also feature two independent DMA engines for simultaneous independent in/out communications. The hardware updates pave the way for LP based GPU communications.
3 LINEAR PIPELINE BASED COLLECTIVE DESIGN DEDICATED FOR NEURAL NETWORK TRAINING ON MULTI-GPUS
This section presents a new LP based MultiGPU collective design ensued by the concrete proof of its performance in training neural networks. The general idea of LP is as follows: a) we dissect a long message into fine-grained blocks. b) a GPU receives a block from the prior GPU via DMA1 while sending a block to the next one via DMA2. Please note each block exchange utilizes an independent physical link, and the entire network is fully utilized once the pipeline is filled.
Broadcast tackles the synchronizations of parameters among multiple GPUs. It copies the source vector to every GPU. Fig.2a illustrates the data flow of the broadcast collective on 3 GPUs. GPU0 is the source, and the rest are destinations. Broadcast starts with filling the pipe by copying block a on GPU0 to GPU1 at step 1. Let’s focus on GPU1. At each step, GPU1 receives a block from GPU0 via DMA1, while GPU1 is also sending a block to GPU2 via DMA2. The data exchange in either way utilizes an independent link and DMA engine to achieve the maximal unidirectional rate. Hence, the bandwidth is fully exploited.
Reduce aggregates the partial gradients to reconstruct the global one. It combines the elements provided in the vector of each GPU, and returns the combined value in the receive vector to a specific GPU. It supports basic arithmetic operations such as summations and multiplications. Fig.2b illustrates the data flow of the reduce collective. GPU2 is the root that aggregates the vectors across all GPUs. Reduce starts with filling the pipe by writing block a0 to a buffer on GPU1. Then, GPU1 reduces the received block a0 with a1 to yield a′ (within the rectangle of Fig.2b). Please note the computation is much faster than the communication, we assume no latency on it. In practice, computations are further overlapped with communications. In the next step, GPU1 retrieves b0 from GPU0 to reduce to b′ via DMA 1, while GPU1 is also sending a′ to GPU2 to reduce to a′′ via DMA 2. b′′, c′′, d′′ are reduced at steps 3, 4, 5 in a similar fashion.
AllReduce enables us to collect partial gradients and broadcast the latest parameters with only one synchronization point per SGD iteration. It combines vectors from all GPUs and distributes the
result back to them. Mathematically, it is equivalent to a reduce followed by a broadcast. However, allreduce is more efficient than two separate calls as it only needs to fill the pipeline once. For example, it takes 9 timesteps to allreduce 4 message blocks, while broadcast + reduce will cost 10. Fig.2c illustrates the data flow of the allreduce collective. It starts with reducing a′′, after which a′′ is broadcast to GPU1 and GPU2 at step 5, 6 respectively. Please note d0 utilizes the outbound DMA at step 4, therefore a′′ has to wait until step 5. b′′, c′′, d′′ are processed in a similar fashion.
Our collective is also specifically designed to accommodate GPU features such as asynchronous kernel launches and multi-stream processing. In the rectangle of Fig.2a, it demonstrates the data transfers are asynchronously launched on two separate streams. The copies happening in the red steps are scheduled on one stream while copies in the black steps are scheduled on another stream. This overlaps the overhead of GPU kernel launches, further improving the pipeline. We illustrate the data flow of the collectives on 3 GPUs. If there are k GPUs, GPU n, 0 < n < k − 1, duplicates the same communication pattern on GPU 1.
3.1 ARCHITECTURE ANALYSIS
LP is the optimal collective algorithm to fully exploit the network bandwidth of a MultiGPU system. Even though PCI-E supports full-duplex communication between any two endpoints, each PCI-E endpoint device only has one input and output port. This results in bandwidth competition if a GPU is receiving from multiple GPUs. Similarly, each PCI-E switch only contains one input and output port used for inter-switch communication, and inter-switch communications of the same direction also compete for the PCI-E bus. It is known that any delay in data movement between two GPUs interrupts the pipelining in the collectives. In such architecture, the communication from parents to children in MST based collective algorithms will compete for the same PCI-E bus, therefore breaking pipelining. The data exchange of BE also suffers from the inter-switch communication congestion in one direction. In contrast, LP connects all GPUs into a chain, and data always flow in one direction. Hence, data movements between two GPUs exclusively occupy the entire PCI-E bus, ensuring uninterrupted pipelining.
3.2 THEORETICAL ANALYSIS
We adopt a cost model widely used by the MPI community to analyze collective operations (Thakur et al. (2005), Thakur & Gropp (2003)). The model assumes the time taken to send a message between two nodes follows:
T = α+ βn+ γn (1)
where α is the latency or startup time of sending a message, β and γ is the transmission rate and reduce rate measured by time per byte, and n is the message size in bytes. We also denote p as the node count, and b as the block size (in bytes) in the pipeline.
Proposition 1 If the network latency α→ 0, Linear Pipeline collectives provide anO(log p) speedup over Minimal Spanning Tree collectives and up to a 2 times speedup over Bidirectional Exchange collectives as the message size n→∞.
Proof. First, we derive the costs of the three Linear Pipeline collectives. According to Fig.2, the length of pipeline is p− 1 + nb blocks assuming each block to be b bytes. A block exchange takes α+ βb+ γb (with reduce) or α+ βb (without reduce). Consequently, broadcast essentially costs (α+βb)(p−1+ nb ) = (p−1+ n b )α+(b(p−1)+n)β, and reduce costs (α+βb+γb)(p−1+ n b ) = (p − 1 + nb )α + (b(p − 1) + n)(β + γ). allreduce is approximately equivalent with a reduce
followed by a broadcast. Therefore, the allreduce’s cost is broadcast’s cost plus reduce’s cost, i.e. 2(p− 1 + nb )α+ (bp− b+ n)(2β + γ). Secondly, we derive the costs of the three Minimal Spanning Tree collectives. MPI adopts MST to broadcast or reduce short messages (Thakur et al. (2005)), the length of which is less than 12 KB. The core concept of MST is to organize p GPUs into a balanced tree of height dlogpe. Then, it takes dlog pe steps to traverse all GPUs in the tree. Each step carries the message of length n, resulting in the cost of broadcast to be the tree height times the cost per step, i.e. log p(α+ nβ) (we omit the ceiling for simplicity). Similarly, MST reduce is log p(α+ nβ + nγ), and MST allreduce is also a combination of broadcast and reduce. Please note the latency term, log pα, is the smallest among algorithms in Table.1, and the bandwidth term, log pnβ, is the slowest as log pnβ nβ. Therefore, MST is widely used for high frequent exchanges of short message.
Finally, we present the costs of the three Bidirectional Exchange collectives. MPI broadcast handles long messages with a MST scatter followed by a BE allgather. Please refer to Chan et al. (2007) for the analysis of BE collectives. Basically, scatter costs ∑dlogpe k=1 (α+2
−knβ) = log pα+ p−1p nβ, while allgather costs (p − 1)α + p−1p nβ. The cost of broadcast is the sum of these two. The MPI long message reduce consists of a reducescatter plus a gather, while allreduce consists of a reducescatter and a allgather. The cost for reducescatter is log pα + p−1p nβ + p−1 p nγ, and both the costs of gather and allgather are log pα + p−1p nβ (also in Chan et al. (2007)). Table 1 summarizes the costs of broadcast, reduce and allreduce for the three different underlying algorithms.
The proposition holds under the assumptions of α → 0 and n → ∞, and these assumptions are legitimate for the training of large scale neural networks on multiGPUs. Nowadays, the PCI Express x16 effectively reduces the latency α down to 10−7s. The current two sockets shared memory machine supports up to 8 GPUs indicating limited p in practice. Let’s take an appropriate block size b to ensure p nb and α n b ∼ 0. This enables us to safely ignore the latency term, e.g. log pα in MST broadcast. On the other hand, current deep convolutional neural network uses a tremendous number of parameters. For example, AlexNet uses 50 MB parameters. The transmission rate1 β ∼ 109Byte/Seconds. Compared to the trivial latency term, the bandwidth term dominates the entire cost T . This result leads us to simplify the costs of BE, MST, and LP based broadcast (Table. 2) to be 2p−1p nβ, nβ log p and (b(p− 1) + n)β, obtaining the following equations:
Tbroadcast BE Tbroadcast LP
≈ 2(1− 1p )
1 + bn (p− 1) < 2 (2)
Tbroadcast MST Tbroadcast LP ≈ log p b(p−1) n + 1 < log p (3)
Compared with broadcast, reduce has the additional γ term. Please note the processing speed of GPUs exceeds TFLOPs implying the term γ ∗ n→ 0. Therefore, it is also legitimate to ignore the γ term, and it yields the same result Treduce BE/Treduce LP < 2 and Treduce MST /Treduce LP < log p. This completes our proof of the proposition 1.
Another interesting point is the cost of Linear Pipeline is invariant to GPU count p regardless of message length n. This implies broadcasting a vector to 8 GPUs should cost the same as broadcasting to 2 GPUs. In practice, we set the block size b around 64 KB, and p is within 101. This suggests the bandwidth term, e.g. the cost of LP broadcast (bp − p + n)β ∼ nβ. Hence, the cost of LP collectives are less likely to be affected by GPU counts p.
3.3 DEEP LEARNING WITH EFFICIENT BSP SGD
We formulate the neural network training as the following optimization problem. Let ψ be a loss function with weight vector w as function parameters that takes randomly sampled images dt as the
1https://en.wikipedia.org/wiki/InfiniBand
Algorithm 1: BSP SGD with communications/computations overlapping. 1 while not converge do 2 broadcast(w0t ) 3 for i ∈ [0, 1, ...,max layers] do 4 nonblocking broadcast(wi+1t ) 5 Forward(i) 6 sync broadcast()
7 Backward(max layers) 8 for i ∈ [max layers− 1, ..., 1, 0] do 9 nonblocking reduce(∇ψi+1sub)
10 Backward(i) 11 sync reduce()
12 wt+1 = GradientUpdate()
Algorithm 2: BSP SGD uses broadcast + reduce. 1 while not converge do 2 ∇ψsub = ForwardBackward(dt) 3 ∇ψ = reduce(∇ψsub) 4 if root then 5 wt+1 = GradientUpdate()
6 broadcast(wt+1) 7 barrier /* sync new w */
Algorithm 3: BSP SGD uses allreduce. 1 while not converge do 2 ∇ψsub = ForwardBackward(dt) 3 ∇ψ = allreduce(∇ψsub) 4 barrier /* collect ∇ψsub */ 5 wt+1 = GradientUpdate() 6 if iter%5 = 0 then 7 broadcast(wt+1)
input. The objective of training is to find an approximate solution to the following problem:
min w
E{ψw(dt)} = ∫
Ω
ψw(dt)dP (4)
A typical neural network training iteration consists of a forward and backward pass. The forward pass yields a loss that measures the discrepancy between the current predictions and the target; The backward pass calculates the gradient, the negative of which points to the steepest descent direction. The gradient descent updates the parameters, w, as follows:
wt = wt−1 − ηt∇ψw(dt) (5)
Guided with Data Parallelism, BSP SGD evenly divides dt into p slices d1t ,d 2 t , ...,d p t so that every GPU computes a partial gradient from dit in parallel. The global gradient is equivalent to the average of partial gradients. After finishing the gradient update, wt is synchronized to all GPUs. We integrate the proposed collectives into this process to harness parallel processing capabilities of multiGPU system. In this paper, we discuss two approaches to BSP SGD implementations.
•fork and join: This approach forks the gradient computations, and joins partial gradients with communications. In this case, communications do not overlap with computations. Alg.2 and Alg.3 demonstrate two collective based implementations using 2 and 1 synchronization points, respectively.
In Alg.2, synchronizations rely on broadcast and reduce. Each GPU calculates a partial gradient referred to as ∇ψsub. The master GPU reconstructs ∇ψ by reducing all ∇ψsub. Then, the GPUs synchronize the latest weight, w, by broadcasting.
In Alg.3, synchronizations only rely on allreduce. The differences between this and Alg.2 are that 1) there is only 1 synchronization point; 2) every GPU computes the gradient update. However, the parameters are not consistent after several iterations due to the precision issues of float multiplications inGradientUpdate. We synchronize w every 5 iterations to enforce consistency while still retaining the benefit of efficient pipelining in allreduce (line 7-8 Alg.3).
•overlapping communications with computations: Another approach is to overlap communications and computations for each network layer. In the forward pass, GPUs broadcast network parameters of layer t+1 during forward computations at layer t. In the backward pass, GPUs reduce
•pros and cons of both approaches: The cost of Alg.2 or Alg.3 is comm+ compt, while the cost of Alg.1 is max(comm, compt). If the network has over a few hundred MB of parameters, the overlapping will be significantly better than the fork and join approach. However, Alg.2 and Alg.3 are relatively easy to implement, and the performance on networks < 100 MB is similar to that of Alg.1.
4 EXPERIMENT
4.1 COLLECTIVES EVALUATION
The MST and BE implementations used in benchmarks are Caffe 2 and OpenMPI. Caffe optimizes the GPU placement in an MST to fully utilize inter-GPU peer to peer (P2P) access. OpenMPI and our implementation, similar to Caffe, also take advantages of P2P. We set up AlexNet and GoogLeNet training using the three BSP SGD algorithms proposed in section 3.3.
Fig.3 presents the performance of LP, MST, and BE based collectives at different message sizes on 4 K40m. The LP broadcast demonstrates an average of 29.2x and 2.3x speedup over BE and MST based alternatives in Caffe and OpenMPI; the LP reduce demonstrates an average of 360.55x and 8.7x speedup over BE and MST reduce, and the LP allreduce demonstrates an average of 109.2x and 7.9x speedup over BE and MST allreduce. In theory, LP is approximately 2x faster than both the MST (p = 4 → logp = 2) and BE approaches. An extraordinary speedup against Open MPI is observable due to inefficient data movement in Open MPI, which moves data to host RAM to perform reduce operations on the CPU before being copied to the target GPU. Instead, we perform reduce on the GPUs, and data blocks directly flow to the target GPU via P2P access. The overlapped reduce computations with communications enables our reduce and allreduce to be 8x faster than that of MST. At each step of MST, GPUs reduce the incoming data only after all the data is available. In contrast, our fine-grained block design enables communications and computations to overlap by reducing a block while receiving a new one in the pipeline. broadcast only involves data copies, and both we and Caffe use P2P to transmit the data. Therefore, the speedup of MST broadcast (2.3x), conforms to the 2.0x theoretical prediction.
The theoretical analysis indicates both the cost of LP and BE collectives are invariant to the GPU count p, while the cost of MST increases with p by a factor of logp. This is also noticeable in the
2Caffe implements an MST based broadcast and reduce for the multiGPU training.
scalability experiment demonstrated in Fig.4. Please note there is a cost jump between 4 and 5 GPUs. Communications have to go through QPI after 4 GPUs incurring the additional cost of copying through the host RAM. The cost of the Linear Pipeline method robustly stays the same if GPU counts =[2,3,4] or [5,6], and QPI explains the inconsistency. The communication steps of MST for 2,3,4,5,6 GPUs are 1,2,2,3,3, respectively. The MST experiments verify the logp cost increase w.r.t GPU counts by evident cost jumps at 3 and 5 GPUs. The data flow of OpenMPI between two GPUs follows GPU RAM→host RAM→GPU RAM. The inefficient data flow inside Open MPI contributes to the near linear cost increase with GPU counts p.
4.2 IMPACT ON THE NEURAL NETWORK TRAINING
Fig.5 demonstrates LP collectives effectively reduce the total training time without affecting SGD’s convergence properties in training large scale neural networks. We use inspurCaffe, Caffe and cuhk’s Caffe branch to benchmark the performance of BE-Alg.1, MST-Alg.1 and BE-Overlap-Alg.3. We also implement Alg.1,2,3, integrated with LP collectives, in Caffe to ensure consistency. Please note the model size affects the communication time, while the batch size affects the computation time. We carefully set these parameters to cover as many cases as possible. Please refer to the captions of Table.2 and Fig.5 for experiment details. We assume these algorithms have similar convergence speeds in iterations as losses of AlexNet are approximately 1 after 30000 iterations and losses of GoogLeNet are approximately 2 after 67000 iterations. However, the time taken to reach the target loss varies dramatically. For example, the speedups of LP-Overlap-Alg.3 over BE-Alg.1 in training AlexNet and GoogLeNet are 2.12x and 2.19x, respectively.
Under Alg.1, but using different underlying collective algorithms, LP-Alg.1 presents 1.91x and 1.74x speedup over BE-Alg.1 and MST-Alg.1 in AlexNet, and 1.6x and 1.1x speedup over BE-Alg.1 and MST-Alg.1 in GoogLeNet. The iteration profiles of these 3 algorithms in Table.2 indicate the communication cost of LP-Alg.1 is only 10% of BE-Alg.1, and 11% of MST-Alg.1 in AlexNet; and 6% of BE-Alg.1, and 43% of MST-Alg.1 in GoogLetNet.
The experiments demonstrate that the speed of the three proposed BSP SGD algorithms is Alg.3 > Alg.2 > Alg.1. The result conforms to our expectations as the cost of Alg.3 is max(comm, compt), while the cost of Alg.1 and Alg.2 is comm+ compt. However, the performance gain is quite limited from Alg.2 to Alg.3 as there is little room left for reducing communications from LP Alg.2 to Alg.3 as demonstrated in Table.2. If the model parameters keep increasing, we expect Alg.3 to be more efficient than Alg.2.

The primary point made by this paper is that given certain architectural characteristics of multi-GPU systems, namely the use of bi-directional PCI-E for communication and the integration of two independent DMA engines on recent GPU devices (providing support for simultaneous independent communications), and given the characteristics of the communications patterns required by synchronous SGD trainers for deep neural networks, namely that the messages are large, dense, and have a fixed length, it makes sense to design communication collectives such as broadcast, reduce, and allreduce specifically for the use case of synchronous SGD training on a multi-GPU system.  The paper describes the implementation of these three collectives (broadcast, reduce, and allreduce) using a linear pipelining (LP) scheme on a (logical) ring topology.  The paper compares the LP collectives to two alternatives:  collectives based on a minimal spanning tree (MST) topology and collectives based on bidirectional exchange (BE).  First, a theoretical comparison is made using a standard cost model used in the high performance computing community.  When assumptions based on multi-GPU system architecture (very low latency for messages) and on the communication characteristics of synchronous SGD training (very large messages) are integrated into the model, the paper finds that the LP collectives should be less costly than BE collectives by a factor of 2 and less costly than MST collectives by a factor of log(p), where p is the number of GPUs being used.  Second, an empirical comparison is performed in which (1) the time required to perform each of the different collectives on a 4-device (k40m) system is measured as a function of message size and (2) the time required to perform each of the different collectives with a 200 MB message length is measured as a function of the number of devices in the system.  These measurements show that the LP-based collectives are consistently the fastest.  Third, DNN training experiments with AlexNet and GoogLeNet are performed on a 4-device system using three different synchronous SGD algorithms with the different implementations of the collectives (a total of 6 different algorithms in all).  Measurements of the communication and computation costs show that the LP collectives reduce communication costs without affecting computation costs (as expected).  Measurements of the convergence of the training loss as a function of time for the two DNN architectures show that use of the LP collectives leads to faster training.
While the theory says that the costs of LP collectives should be invariant to the number of devices in a multi-GPU system, the empirical work shows that in practice this does not hold going from 4 to 5 devices (in the tested configuration) because in a 5-device system messages must traverse the QPI.  Are there other practical considerations that the authors are aware of that affect the scaling of the LP collectives?  If so, these should be mentioned in the paper.
In the sentence "Worringen (2003) proposed a pipeline collective model in shared memory environment for CPU data, but communications of different MPI processes sharing the same CPU memory bus within the same CPU socket." I really can't figure out what the words after "but communications of different MPI processes" are trying to convey.  This sentence is not comprehensible.
"Please note the latency term is log pα, which is the smallest among algorithms in Table.1. Therefore, MST only suits for high frequent short messages."  The claim that MST collectives are only suitable for high-frequency, short messages does not follow from the statement that MST collectives have the smallest latency term.  You also need to consider the way the cost scales with message size (the bandwidth term).  If the MST collectives had a better bandwidth term than the other collectives, then they would also be superior for large messages.
"Let’s take an appropriate block size b to ensure n/b ≪ α."  This looks wrong, since n > b.  Should it be b/n ≪ α?
"However, the parameters are not consistent after several iterations due to the precision issues of float multiplications in Gradient Update."  Are you sure the inconsistency in weight estimates across devices is due to multiplication?  I would expect that it would be due to gradients being accumulated in different orders; that is, because floating point addition is not commutative.
I recommend replacing the term "sub-gradients" in this paper with "partial gradients."  In the optimization literature, the term "sub-gradient" has a very specific meaning that differs from this paper's use of the term (see


DROPOUT WITH EXPECTATION-LINEAR REGULARIZATION
Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout’s training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.
1 INTRODUCTION
Deep neural networks (DNNs, e.g., LeCun et al., 2015; Schmidhuber, 2015), if trained properly, have been demonstrated to significantly improve the benchmark performances in a wide range of application domains. As neural networks go deeper and deeper, naturally, its model complexity also increases quickly, hence the pressing need to reduce overfitting in training DNNs. A number of techniques have emerged over the years to address this challenge, among which dropout (Hinton et al., 2012; Srivastava, 2013) has stood out for its simplicity and effectiveness. In a nutshell, dropout randomly “drops” neural units during training as a means to prevent feature co-adaptation—a sign of overfitting (Hinton et al., 2012). Simple as it appears to be, dropout has led to several record-breaking performances (Hinton et al., 2012; Ma & Hovy, 2016), and thus spawned a lot of recent interests in analyzing and justifying dropout from the theoretical perspective, and also in further improving dropout from the algorithmic and practical perspective.
In their pioneering work, Hinton et al. (2012) and Srivastava et al. (2014) interpreted dropout as an extreme form of model combination (aka. model ensemble) with extensive parameter/weight sharing, and they proposed to learn the combination through minimizing an appropriate expected loss. Interestingly, they also pointed out that for a single logistic neural unit, the output of dropout is in fact the geometric mean of the outputs of the model ensemble with shared parameters. Subsequently, many theoretical justifications of dropout have been explored, and we can only mention a few here due to space limits. Building on the weight sharing perspective, Baldi & Sadowski (2013; 2014) analyzed the ensemble averaging property of dropout in deep non-linear logistic networks, and supported the view that dropout is equivalent to applying stochastic gradient descent on some regularized
loss function. Wager et al. (2013) treated dropout as an adaptive regularizer for generalized linear models (GLMs). Helmbold & Long (2016) discussed the differences between dropout and traditional weight decay regularization. In terms of statistical learning theory, Gao & Zhou (2014) studied the Rademacher complexity of different types of dropout, showing that dropout is able to reduce the Rademacher complexity polynomially for shallow neural networks (with one or no hidden layers) and exponentially for deep neural networks. This latter work (Gao & Zhou, 2014) formally demonstrated that dropout, due to its regularizing effect, contributes to reducing the inherent model complexity, in particular the variance component in the generalization error.
Seen as a model combination technique, it is intuitive that dropout contributes to reducing the variance of the model performance. Surprisingly, dropout has also been shown to play some role in reducing the model bias. For instance, Jain et al. (2015) studied the ability of dropout training to escape local minima, hence leading to reduced model bias. Other studies (Chen et al., 2014; Helmbold & Long, 2014; Wager et al., 2014) focus on the effect of the dropout noise on models with shallow architectures. We noted in passing that there are also some work (Kingma et al., 2015; Gal & Ghahramani, 2015; 2016) trying to understand dropout from the Bayesian perspective.
In this work, we first formulate dropout as a tractable approximation of a latent variable model, and give a clean view of weight sharing (§3). Then, we focus on an inference gap in dropout that has somehow gotten under-appreciated: In the inference phase, for computational tractability considerations, the model ensemble generated by dropout is approximated by a single model with scaled weights, resulting in a gap between training and inference, and rendering the many previous theoretical findings inapplicable. In general, this inference gap can be very large and no attempt (to our best knowledge) has been made to control it. We make three contributions in bridging this gap: Theoretically, we introduce expectation-linear dropout neural networks, through which we are able to explicitly quantify the inference gap (§4). In particular, our theoretical results explain why the max-norm constraint on the network weights, a standard practice in training DNNs, can lead to a small inference gap hence potentially improve performance. Algorithmically, we propose to add a sampled version of the inference gap to regularize the standard dropout training objective (expectationlinearization), hence allowing explicit control of the inference gap, and analyze the interaction between expectation-linearization and the model accuracy (§5). Experimentally, through three benchmark datasets we show that our regularized dropout is not only as simple and efficient as standard dropout but also consistently leads to improved performance (§6).
2 DROPOUT NEURAL NETWORKS
In this section we set up the notations, review the dropout neural network model, and discuss the inference gap in standard dropout training that we will attempt to study in the rest of the paper.
2.1 DNNS AND NOTATIONS
Throughout we use uppercase letters for random variables (and occasionally for matrices as well), and lowercase letters for realizations of the corresponding random variables. Let X ∈ X be the input of the neural network, Y ∈ Y be the desired output, and D = {(x1, y1), . . . , (xN , yN )} be our training sample, where xi, i = 1, . . . , N, (resp. yi) are usually i.i.d. samples of X (resp. Y ).
Let M denote a deep neural network with L hidden layers, indexed by l ∈ {1, . . . , L}. Let h(l) denote the output vector from layer l. As usual, h(0) = x is the input, and h(L) is the output of the neural network. Denote θ = {θl : l = 1, . . . , L} as the set of parameters in the network M, where θl assembles the parameters in layer l. With dropout, we need to introduce a set of dropout random variables S = {Γ(l) : l = 1, . . . , L}, where Γ(l) is the dropout random variable for layer l. Then the deep neural network M can be described as:
h(l) = fl(h (l−1) γ(l); θl), l = 1, . . . , L, (1)
where is the element-wise product and fl is the transformation function of layer l. For example, if layer l is a fully connected layer with weight matrix W , bias vector b, and sigmoid activation function σ(x) = 11+exp(−x) , then fl(x) = σ(Wx+ b)). We will also use h
(l)(x, s; θ) to denote the output of layer l with input x and dropout value s, under parameter θ.
In the simplest form of dropout, which is also called standard dropout, Γ(l) is a vector of independent Bernoulli random variables, each of which has probability pl of being 1 and 1− pl of being 0. This corresponds to dropping each of the weights independently with probability pl.
2.2 DROPOUT TRAINING
The standard dropout neural networks can be trained using stochastic gradient decent (SGD), with a sub-network sampled by dropping neural units for each training instance in a mini-batch. Forward and backward pass for that training instance are done only on the sampled sub-network. Intuitively, dropout aims at, simultaneously and jointly, training an ensemble of exponentially many neural networks (one for each configuration of dropped units) while sharing the same weights/parameters.
The goal of the stochastic training procedure of dropout can be understood as minimizing an expected loss function, after marginalizing out the dropout variables (Srivastava, 2013; Wang & Manning, 2013). In the context of maximal likelihood estimation, dropout training can be formulated as:
θ∗ = argmin θ ESD [−l(D,SD; θ)] = argmin θ ESD
[ − N∑ i=1 log p(yi|xi, Si; θ) ] , (2)
where recall that D is the training sample, SD = {S1, . . . , SN} is the dropout variable (one for each training instance), and l(D,SD; θ) is the (conditional) log-likelihood function defined by the conditional distribution p(y|x, s; θ) of output y given input x, under parameter θ and dropout variable s. Throughout we use the notation EZ to denote the conditional expectation where all random variables except Z are conditioned on.
Dropout has also been shown to work well with regularization, such as L2 weight decay (Tikhonov, 1943), Lasso (Tibshirani, 1996), KL-sparsity(Bradley & Bagnell, 2008; Hinton, 2010), and max-norm regularization (Srebro et al., 2004), among which the max-norm regularization — that constrains the norm of the incoming weight matrix to be bounded by some constant — was found to be especially useful for dropout (Srivastava, 2013; Srivastava et al., 2014).
2.3 DROPOUT INFERENCE AND GAP
As mentioned before, dropout is effectively training an ensemble of neural networks with weight sharing. Consequently, at test time, the output of each network in the ensemble should be averaged to deliver the final prediction. This averaging over exponentially many sub-networks is, however, intractable, and standard dropout typically implements an approximation by introducing a deterministic scaling factor for each layer to replace the random dropout variable:
ES [H (L)(x, S; θ)] ? ≈ h(L)(x,E[S]; θ), (3)
where the right-hand side is the output of a single deterministic neural network whose weights are scaled to match the expected number of active hidden units on the left-hand side. Importantly, the right-hand side can be easily computed since it only involves a single deterministic network.
Bulò et al. (2016) combined dropout with knowledge distillation methods (Hinton et al., 2015) to better approximate the averaging processing of the left-hand side. However, the quality of the approximation in (3) is largely unknown, and to our best knowledge, no attempt has been made to explicitly control this inference gap. The main goal of this work is to explicitly quantify, algorithmically control, and experimentally demonstrate the inference gap in (3), in the hope of improving the generalization performance of DNNs eventually. To this end, in the next section we first present a latent variable model interpretation of dropout, which will greatly facilitate our later theoretical analysis.
3 DROPOUT AS LATENT VARIABLE MODELS
With the end goal of studying the inference gap in (3) in mind, in this section, we first formulate dropout neural networks as a latent variable model (LVM) in § 3.1. Then, we point out the relation between the training procedure of LVM and that of standard dropout in § 3.2. The advantage of formulating dropout as a LVM is that we need only deal with a single model (with latent structure), instead of an ensemble of exponentially many different models (with weight sharing). This much
simplified view of dropout enables us to understand and analyze the model parameter θ in a much more straightforward and intuitive way.
3.1 AN LVM FORMULATION OF DROPOUT
A latent variable model consists of two types of variables: the observed variables that represent the empirical (observed) data and the latent variables that characterize the hidden (unobserved) structure. To formulate dropout as a latent variable model, the input x and output y are regarded as observed variables, while the dropout variable s, representing the sub-network structure, is hidden. Then, upon fixing the input space X , the output space Y , and the latent space S for dropout variables, the conditional probability of y given x under parameter θ can be written as
p(y|x; θ) = ∫ S p(y|x, s; θ)p(s)dµ(s), (4)
where p(y|x, s; θ) is the conditional distribution modeled by the neutral network with configuration s (same as in Eq. (2)), p(s) is the distribution of dropout variable S (e.g. Bernoulli), here assumed to be independent of the input x, and µ(s) is the base measure on the space S.
3.2 LVM DROPOUT TRAINING VS. STANDARD DROPOUT TRAINING
Building on the above latent variable model formulation (4) of dropout, we are now ready to point out a simple relation between the training procedure of LVM and that of standard dropout. Given an i.i.d. training sample D, the maximum likelihood estimate for the LVM formulation of dropout in (4) is equivalent to minimizing the following negative log-likelihood function:
θ∗ = argmin θ −l(D; θ) = argmin θ − N∑ i=1 log p(yi|xi; θ), (5)
where p(y|x; θ) is given in Eq. (4). Recall the dropout training objective ESD [−l(D,SD; θ)] in Eq. (2). We have the following theorem as a simple consequence of Jensen’s inequality (details in Appendix A): Theorem 1. The expected loss function of standard dropout (Eq. (2)) is an upper bound of the negative log-likelihood of LVM dropout (Eq. (5)):
−l(D; θ) ≤ ESD [−l(D,SD; θ)]. (6)
Theorem 1, in a rigorous sense, justifies dropout training as a convenient and tractable approximation of the LVM formulation in (4). Indeed, since directly minimizing the marginalized negative loglikelihood in (5) may not be easy, a standard practice is to replace the marginalized (conditional) likelihood p(y|x; θ) in (4) with its empirical Monte carlo average through drawing samples from the dropout variable S. The dropout training objective in (2) corresponds exactly to this Monte carlo approximation when a single sample Si is drawn for each training instance (xi, yi). Importantly, we note that the above LVM formulation involves only a single network parameter θ, which largely simplifies the picture and facilitates our subsequent analysis.
4 EXPECTATION-LINEAR DROPOUT NEURAL NETWORKS
Building on the latent variable model formulation in § 3, we introduce in this section the notion of expectation-linearity that essentially measures the inference gap in (3). We then characterize a general class of neural networks that exhibit expectation-linearity, either exactly or approximately over a distribution p(x) on the input space.
We start with defining expectation-linearity in the simplest single-layer neural network, then we extend the notion into general deep networks in a natural way. Definition 1 (Expectation-linear Layer). A network layer h = f(x γ; θ) is expectation-linear with respect to a set X ′ ⊆ X , if for all x ∈ X ′ we have∥∥E[f(x Γ; θ)]− f(x E[Γ]; θ)∥∥
2 = 0. (7)
In this case we say that X ′ is expectation-linearizable, and θ is expectation-linearizing w.r.t X ′.
Obviously, the condition in (7) will guarantee no gap in the dropout inference approximation (3)—an admittedly strong condition that we will relax below. Clearly, if f is an affine function, then we can choose X ′ = X and expectation-linearity is trivial. Note that expectation-linearity depends on the network parameter θ and the dropout distribution Γ.
Expectation-linearity, as defined in (7), is overly strong: under standard regularity conditions, essentially the transformation function f has to be affine over the set X ′, ruling out for instance the popular sigmoid or tanh activation functions. Moreover, in practice, downstream use of DNNs are usually robust to small errors resulting from approximate expectation-linearity (hence the empirical success of dropout), so it makes sense to define an inexact extension. We note also that the definition in (7) is uniform over the set X ′, while in a statistical setting it is perhaps more meaningful to have expectation-linearity “on average,” since inputs from lower density regions are not going to play a significant role anyway. Taking into account the aforementioned motivations, we arrive at the following inexact extension: Definition 2 (Approximately Expectation-linear Layer). A network layer h = f(x γ; θ) is δ-approximately expectation-linear with respect to a distribution p(x) over X if
EX [∥∥EΓ[f(X Γ; θ)|X]− f(X E[Γ]; θ)∥∥2] < δ. (8) In this case we say that p(x) is δ-approximately expectation-linearizable, and θ is δ-approximately expectation-linearizing.
To appreciate the power of cutting some slack from exact expectation-linearity, we remark that even non-affine activation functions often have approximately linear regions. For example, the logistic function, a commonly used non-linear activation function in DNNs, is approximately linear around the origin. Naturally, we can ask whether it is sufficient for a target distribution p(x) to be well-approximated by an approximately expectation-linearizable one. We begin by providing an appropriate measurement of the quality of this approximation. Definition 3 (Closeness, (Andreas et al., 2015)). A distribution p(x) is C-close to a set X ′ ⊆ X if
E [
inf x∗∈X ′ sup γ∈S ‖X γ − x∗ γ‖2
] ≤ C, (9)
where recall that S is the (bounded) space that the dropout variable lives in.
Intuitively, p(x) is C-close to a set X ′ if a random sample from p is no more than a distance C from X ′ in expectation and under the worst “dropout perturbation”. For example, a standard normal distribution is close to an interval centering at origin ([−α, α]) with some constant C. Our definition of closeness is similar to that in Andreas et al. (2015), who used this notion to analyze self-normalized log-linear models.
We are now ready to state our first major result that quantifies approximate expectation-linearity of a single-layered network (proof in Appendix B.1): Theorem 2. Given a network layer h = f(x γ; θ), where θ is expectation-linearizing w.r.t. X ′ ⊆ X . Suppose p(x) is C-close to X ′ and for all x ∈ X , ‖∇xf(x)‖op ≤ B, where ‖ · ‖op is the usual operator norm. Then, p(x) is 2BC-approximately expectation-linearizable.
Roughly, Theorem 2 states that the input distribution p(x) that place most of its mass on regions close to expectation-linearizable sets are approximately expectation-linearizable on a similar scale. The bounded operator norm assumption on the derivative ∇f is satisfied in most commonly used layers. For example, for a fully connected layer with weight matrix W , bias vector b, and activation function σ, ‖∇f(·)‖op = |σ′(·)| · ‖W‖op is bounded by ‖W‖op and the supremum of |σ′(·)| (1/4 when σ is sigmoid and 1 when σ is tanh).
Next, we extend the notion of approximate expectation-linearity to deep dropout neural networks. Definition 4 (Approximately Expectation-linear Network). A deep neural network with L layers (cf. Eq. (1)) is δ-approximately expectation-linear with respect to p(x) over X if
EX [∥∥ES[H(L)(X,S; θ)|X]− h(L)(X,E[S]; θ)∥∥2] < δ. (10) where h(L)(X,E[S]; θ) is the output of the deterministic neural network in standard dropout.
Lastly, we relate the level of approximate expectation-linearity of a deep neural network to the level of approximate expectation-linearity of each of its layers: Theorem 3. Given an L-layer neural network as in Eq. (1), and suppose that each layer l ∈ {1, . . . , L} is δ-approximately expectation-linear w.r.t. p(h(l)), E[Γ(l)] ≤ γ, supx ‖∇fl(x)‖op ≤ B, and E [ Var[H(l)|X] ] ≤ σ2. Then the network is ∆-approximately expectation-linear with
∆ = (Bγ)L−1δ + (δ +Bγσ)
( 1− (Bγ)L−1
1−Bγ
) . (11)
From Theorem 3 (proof in Appendix B.2) we observe that the level of approximate expectationlinearity of the network mainly depends on four factors: the level of approximate expecatationlinearity of each layer (δ), the expected variance of each layer (σ), the operator norm of the derivative of each layer’s transformation function (B), and the mean of each layer’s dropout variable (γ). In practice, γ is often a constant less than or equal to 1. For example, if Γ ∼ Bernoulli(p), then γ = p. According to the theorem, the operator norm of the derivative of each layer’s transformation function is an important factor in the level of approximate expectation-linearity: the smaller the operator norm is, the better the approximation. Interestingly, the operator norm of a layer often depends on the norm of the layer’s weight (e.g. for fully connected layers). Therefore, adding max-norm constraints to regularize dropout neural networks can lead to better approximate expectation-linearity hence smaller inference gap and the often improved model performance.
It should also be noted that when Bγ < 1, the approximation error ∆ tends to be a constant when the network becomes deeper. When Bγ = 1, ∆ grows linearly with L, and when Bγ > 1, the growth of ∆ becomes exponential. Thus, it is essential to keep Bγ < 1 to achieve good approximation, particularly for deep neural networks.
5 EXPECTATION-LINEAR REGULARIZED DROPOUT
In the previous section we have managed to bound the approximate expectation-linearity, hence the inference gap in (3), of dropout neural networks. In this section, we first prove a uniform deviation bound of the sampled approximate expectation-linearity measure from its mean, which motivates adding the sampled (hence computable) expectation-linearity measure as a regularization scheme to standard dropout, with the goal of explicitly controlling the inference gap of the learned parameter, hence potentially improving the performance. Then we give the upper bounds on the loss in accuracy due to expectation-linearization, and describe classes of distributions that expectation-linearize easily.
5.1 A UNIFORM DEVIATION BOUND FOR THE SAMPLED EXPECTATION-LINEAR MEASURE
We now show that an expectation-linear network can be found by expectation-linearizing the network on the training sample. To this end, we prove a uniform deviation bound between the empirical expectation-linearization measure using i.i.d. samples (Eq. (12)) and its mean (Eq. (13)).
Theorem 4. Let H = { h(L)(x, s; θ) : θ ∈ Θ } denote a space of L-layer dropout neural networks
indexed with θ, where h(L) : X × S → R and Θ is the space that θ lives in. Suppose that the neural networks inH satisfy the constraints: 1) ∀x ∈ X , ‖x‖2 ≤ α; 2) ∀l ∈ {1, . . . , L},E(Γ(l)) ≤ γ and ‖∇fl‖op ≤ B; 3) ‖h(L)‖ ≤ β. Denote empirical expectation-linearization measure and its mean as:
∆̂ = 1
n n∑ i=1 ∥∥ESi[H(L)(Xi, Si; θ)]− h(L)(Xi,E[Si]; θ)∥∥2, (12) ∆ = EX
[∥∥ES[H(L)(X,S; θ)]− h(L)(X,E[S]; θ)∥∥2]. (13) Then, with probability at least 1− ν, we have
sup θ∈Θ |∆− ∆̂| < 2αB L(γL/2 + 1)√ n + β
√ log(1/ν)
n . (14)
From Theorem 4 (proof in Appendix C.1) we observe that the deviation bound decreases exponentially with the number of layers L when the operator norm of the derivative of each layer’s transformation
function (B) is less than 1 (and the contrary if B ≥ 1). Importantly, the square root dependence on the number of samples (n) is standard and cannot be improved without significantly stronger assumptions.
It should be noted that Theorem 4 per se does not imply anything between expectation-linearization and the model accuracy (i.e. how well the expectation-linearized neural network actually achieves on modeling the data). Formally studying this relation is provided in § 5.3. In addition, we provide some experimental evidences in § 6 on how improved approximate expectation-linearity (equivalently smaller inference gap) does lead to better empirical performances.
5.2 EXPECTATION-LINEARIZATION AS REGULARIZATION
The uniform deviation bound in Theorem 4 motivates the possibility of obtaining an approximately expectation-linear dropout neural networks through adding the empirical measure (12) as a regularization scheme to the standard dropout training objective, as follows:
loss(D; θ) = −l(D; θ) + λV (D; θ), (15) where −l(D; θ) is the negative log-likelihood defined in Eq. (5), λ > 0 is a regularization constant, and V (D; θ) measures the level of approximate expectation-linearity:
V (D; θ) = 1
N N∑ i=1 ∥∥ESi[H(L)(xi, Si; θ)]− h(L)(xi,E[Si]; θ)∥∥22. (16) To solve (15), we can minimize loss(D; θ) via stochastic gradient descent as in standard dropout, and approximate V (D; θ) using Monte carlo:
V (D; θ) ≈ 1 N N∑ i=1 ∥∥h(L)(xi, si; θ)− h(L)(xi,E[Si]; θ)∥∥22, (17) where si is the same dropout sample as in l(D; θ) for each training instance in a mini-batch. Thus, the only additional computational cost comes from the deterministic term h(L)(xi,E[Si]; θ). Overall, our regularized dropout (15), in its Monte carlo approximate form, is as simple and efficient as the standard dropout.
5.3 ON THE ACCURACY OF EXPECTATION-LINEARIZED MODELS
So far our discussion has concentrated on the problem of finding expectation-linear neural network models, without any concerns on how well they actually perform at modeling the data. In this section, we characterize the trade-off between maximizing “data likelihood” and satisfying an expectationlinearization constraint.
To achieve the characterization, we measure the likelihood gap between the classical maximum likelihood estimator (MLE) and the MLE subject to a expectation-linearization constraint. Formally, given training data D = {(x1, y1), . . . , (xn, yn)}, we define
θ̂ = argmin θ∈Θ
−l(D; θ) (18)
θ̂δ = argmin θ∈Θ,V (D;θ)≤δ
−l(D; θ) (19)
where −l(D; θ) is the negative log-likelihood defined in Eq. (5), and V (D; θ) is the level of approximate expectation-linearity in Eq. (16).
We would like to control the loss of model accuracy by obtaining a bound on the likelihood gap defined as:
∆l(θ̂, θ̂δ) = 1
n (l(D; θ̂)− l(D; θ̂δ)) (20)
In the following, we focus on neural networks with softmax output layer for classification tasks.
p(y|x, s; θ) = h(L)y (x, s; θ) = fL(h(L−1)(x, s); η) = eη
T y h (L−1)(x,s)∑ y′∈Y e ηT y′h (L−1)(x,s) (21)
where θ = {θ1, . . . , θL−1, η}, Y = {1, . . . , k} and η = {ηy : y ∈ Y}. We claim:
Theorem 5. Given an L-layer neural network h(L)(x, s; θ) with softmax output layer in (21), where parameter θ ∈ Θ, dropout variable s ∈ S, input x ∈ X and target y ∈ Y . Suppose that for every x and s, p(y|x, s; θ̂) makes a unique best prediction—that is, for each x ∈ X , s ∈ S, there exists a unique y∗ ∈ Y such that ∀y 6= y∗, η̂Ty h(L−1)(x, s) < η̂Ty∗h(L−1)(x, s). Suppose additionally that ∀x, s, ‖h(L−1)(x, s; θ̂)‖ ≤ β, and ∀y, p(y|x; θ̂) > 0. Then
∆l(θ̂, θ̂δ) ≤ c1β2 ( ‖η̂‖2 − δ
4β
)2 e−c2δ/4β (22)
where c1 and c2 are distribution-dependent constants.
From Theorem 5 (proof in Appendix C.2) we observe that, at one extreme, distributions closed to deterministic can be expectation-linearized with little loss of likelihood.
What about the other extreme — distributions “as close to uniform distribution as possible”? With suitable assumptions about the form of p(y|x, s; θ̂) and p(y|x; θ̂), we can achieve an accuracy loss bound for distributions that are close to uniform: Theorem 6. Suppose that ∀x, s, ‖h(L−1)(x, s; θ̂)‖ ≤ β. Additionally, for each (xi, yi) ∈ D, s ∈ S , log 1k ≤ log p(yi|xi, s; θ̂) ≤ 1 k ∑ y∈Y log p(y|xi, s; θ̂). Then asymptotically as n→∞:
∆l(θ̂, θ̂δ) ≤ (
1− δ 4β‖η̂‖2
) E [KL (p(·|X; θ)‖Unif(Y))] (23)
Theorem 6 (proof in Appendix C.3) indicates that uniform distributions are also an easy class for expectation-linearization.
The next question is whether there exist any classes of conditional distributions p(y|x) for which all distributions are provably hard to expectation-linearize. It remains an open problem and might be an interesting direction for future work.
6 EXPERIMENTS
In this section, we evaluate the empirical performance of the proposed regularized dropout in (15) on a variety of network architectures for the classification task on three benchmark datasets—MNIST, CIFAR-10 and CIFAR-100. We applied the same data preprocessing procedure as in Srivastava et al. (2014). To make a thorough comparison and provide experimental evidence on how the expectationlinearization interacts with the predictive power of the learned model, we perform experiments of Monte Carlo (MC) dropout, which approximately computes the final prediction (left-hand side of (3)) via Monte Carlo sampling, w/o the proposed regularizer. In the case of MC dropout, we average m = 100 predictions using randomly sampled configurations. In addition, the network architectures and hyper-parameters for each experiment setup are the same as those in Srivastava et al. (2014), unless we explicitly claim to use different ones. Following previous works, for each data set We held out 10,000 random training images for validation to tune the hyper-parameters, including λ in Eq. (15). When the hyper-parameters are fixed, we train the final models with all the training data, including the validation data. A more detailed description of the conducted experiments can be provided in Appendix D. For each experiment, we report the mean test errors with corresponding standard deviations over 5 repetitions.
6.1 MNIST
The MNIST dataset (LeCun et al., 1998) consists of 70,000 handwritten digit images of size 28×28, where 60,000 images are used for training and the rest for testing. This task is to classify the images into 10 digit classes. For the purpose of comparison, we train 6 neural networks with different architectures. The experimental results are shown in Table 1.
6.2 CIFAR-10 AND CIFAR-100
The CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) consist of 60,000 color images of size 32× 32, drawn from 10 and 100 categories, respectively. 50,000 images are used for training and the
rest for testing. The neural network architecture we used for these two datasets has 3 convolutional layers, followed by two fully-connected (dense) hidden layers (again, same as that in Srivastava et al. (2014)). The experimental results are recorded in Table 1, too.
From Table 1 we can see that on MNIST data, dropout network training with expectation-linearization outperforms standard dropout on all 6 neural architectures. On CIFAR data, expectation-linearization reduces error rate from 12.82% to 12.20% for CIFAR-10, achieving 0.62% improvement. For CIFAR-100, the improvement in terms of error rate is 0.97% with reduction from 37.22% to 36.25%.
From the results we see that with or without expectation-linearization, the MC dropout networks achieve similar results. It illustrates that by achieving expectation-linear neural networks, the predictive power of the learned models has not degraded significantly. Moreover, it is interesting to see that with the regularization, on MNIST dataset, standard dropout networks achieve even better accuracy than MC dropout. It may be because that with expectation-linearization, standard dropout inference achieves better approximation of the final prediction than MC dropout with (only) 100 samples. On CIFAR datasets, MC dropout networks achieve better accuracy than the ones with the regularization. But, obviously, MC dropout requires much more inference time than standard dropout (MC dropout with m samples requires about m times the inference time of standard dropout).
6.3 EFFECT OF REGULARIZATION CONSTANT λ
In this section, we explore the effect of varying the hyper-parameter for the expectation-linearization rate λ. We train the network architectures in Table 1 with the λ value ranging from 0.1 to 10.0. Figure 1 shows the test errors obtained as a function of λ on three datasets. In addition, Figure 1, middle and right panels, also measures the empirical expectation-linearization risk ∆̂ of Eq. (12) with varying λ on CIFAR-10 and CIFAR-100, where ∆̂ is computed using Monte carlo with 100 independent samples.
From Figure 1 we can see that when λ increases, better expectation-linearity is achieved (i.e. ∆̂ decreases). The model accuracy, however, has not kept growing with increasing λ, showing that in practice considerations on the trade-off between model expectation-linearity and accuracy are needed.
6.4 COMPARISON WITH DROPOUT DISTILLATION
To make a thorough empirical comparison with the recently proposed Dropout Distillation method (Bulò et al., 2016), we also evaluate our regularization method on CIFAR-10 and CIFAR-100 datasets with the All Convolutional Network (Springenberg et al., 2014) (AllConv). To facilitate comparison, we adopt the originally reported hyper-parameters and the same setup for training.
Table 2 gives the results comparison the classification error percentages on test data under AllConv using standard dropout, Monte Carlo dropout, standard dropout with our proposed expectationlinearization, and recently proposed dropout distillation on CIFAR-10 and CIFAR-100 1. According to Table 2, our proposed expectation-linear regularization method achieves comparable performance to dropout distillation.
7 CONCLUSIONS
In this work, we attempted to establish a theoretical basis for the understanding of dropout, motivated by controlling the gap between dropout’s training and inference phases. Through formulating dropout as a latent variable model and introducing the notion of (approximate) expectation-linearity, we have formally studied the inference gap of dropout, and introduced an empirical measure as a regularization scheme to explicitly control the gap. Experiments on three benchmark datasets demonstrate that reducing the inference gap can indeed improve the end performance. In the future, we intend to formally relate the inference gap to the generalization error of the underlying network, hence providing further justification of regularized dropout.
ACKNOWLEDGEMENTS
This research was supported in part by DARPA grant FA8750-12-2-0342 funded under the DEFT program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA.
APPENDIX: DROPOUT WITH EXPECTATION-LINEAR REGULARIZATION
A LVM DROPOUT TRAINING VS. STANDARD DROPOUT TRAINING
Proof of Theorem 1
Proof.
ESD [l(D,SD; θ)] = ∫ S N∏ i=1 p(si) ( N∑ i=1 log p(yi|xi, si; θ) ) dµ(s1) . . . dµ(sN )
= N∑ i=1 ∫ S p(si) log p(yi|xi, si; θ)dµ(si)
Because log(·) is a concave function, from Jensen’s Inequality,∫ S p(s) log p(y|x, s; θ)dµ(s) ≤ log ∫ S p(s)p(y|x, s; θ)dµ(s)
Thus
ESD [−l(D,SD; θ)] ≥ N∑ i=1 log ∫ S p(si)p(yi|xi, si; θ)dµ(si) = −l(D; θ).
B EXPECTATION-LINEAR DROPOUT NEURAL NETWORKS
B.1 PROOF OF THEOREM 2
Proof. Let γ∗ = E[Γ], and
A ∆ = {x : ‖E[f(x Γ; θ)]− f(x γ∗; θ)‖2 = 0}
Let X∗ = argmin x∈A sup γ∈S ‖X γ − x γ‖2, and X− = X −X∗. Then,
X γ = X∗ γ +X− γ
In the following, we omit the parameter θ for convenience. Moreover, we denote EΓ [ f(X Γ; θ) ] ∆ = E [ f(X Γ; θ)|X ] From Taylor Series, there exit some X ′, X ′′ ∈ X satisfy that
f(X Γ) = f(X∗ Γ) + f ′(X ′ Γ)(X− Γ) f(X γ∗) = f(X∗ γ∗) + f ′(X ′′ γ∗)(X− γ∗)
where we denote f ′(x) = (∇xf(x))T . Then,
EΓ[f(X Γ)− f(X γ∗)] = EΓ[f(X
∗ Γ +X− Γ)− f(X∗ γ∗ +X− γ∗)] = EΓ[f(X
∗ Γ)− f(X∗ γ∗) + f ′(X ′ Γ)(X− Γ)− f ′(X ′′ γ∗)(X− γ∗)] = EΓ[f(X ∗ Γ)− f(X∗ γ∗)] + EΓ[f ′(X ′ Γ)(X− Γ)− f ′(X ′′ γ∗)(X− γ∗)]
Since X∗ ∈ A, we have EΓ[f(X
∗ Γ)− f(X∗ γ∗)] = 0. Then,
EΓ[f(X Γ)− f(X γ∗)] = EΓ[f
′(X ′ Γ)(X− Γ)− f ′(X ′′ γ∗)(X− γ∗)] = EΓ[(f
′(X ′ Γ)− f ′(X ′′ γ∗))(X− Γ)] + EΓ[f ′(X ′′ γ∗)(X− Γ−X− γ∗)] = EΓ[(f ′(X ′ Γ)− f ′(X ′′ γ∗))(X− Γ)]
Then, ‖EΓ[f(X Γ)]− f(X γ∗)‖2 = ‖EΓ[(f ′(X ′ Γ)− f ′(X ′′ γ∗))(X− Γ)]‖2 Since ‖X− γ′‖2 ≤ sup
γ∈S ‖X− γ‖2 = inf x∈A sup γ∈S ‖X γ − x γ‖2, and from Jensen’s inequality
and property of operator norm,
‖EΓ[(f ′(X ′ Γ)− f ′(X ′′ γ∗))(X− Γ)]‖2 ≤ EΓ [ ‖f ′(X ′ Γ)− f ′(X ′′ γ∗)‖op‖X− Γ‖2 ] ≤ 2BEΓ [ ‖X− Γ‖2
] ≤ 2B inf
x∈A sup γ∈S ‖X γ − x γ‖2
Finally we have,
EX [ ‖EΓ[(f ′(X ′ Γ)− f ′(X ′′ γ∗))(X− Γ)]‖2 ] ≤ 2BE [ inf x∈A sup γ∈S ‖X γ − x γ‖2 ] ≤ 2BC
B.2 PROOF OF THEOREM 3
Proof. Induction on the number of the layers L. As before, we omit the parameter θ. Initial step: when L = 1, the statement is obviously true. Induction on L: Suppose that the statement is true for neural networks with L layers. Now we prove the case L+ 1. From the inductive assumption, we have,
EX [∥∥ESL[H(L)(X,SL)]− h(L)(X,E[SL])∥∥2] ≤ ∆L (1) where SL = {Γ(1), . . . ,Γ(L)} is the dropout random variables for the first L layers, and
∆L = (Bγ) L−1δ + (δ +Bγσ)
( 1− (Bγ)L−1
1−Bγ ) In addition, the L+ 1 layer is δ-approximately expectation-linear, we have:
EH(L) [∥∥EΓ(L+1)[fL+1(H(L) Γ(L+1))]− fL+1(H(L) γ(L+1))∥∥2] ≤ δ (2)
Let E[Γ(l)] = γ(l),∀l ∈ {1, . . . , L + 1}, and let H(l) and h(l) be short for H(l)(X,Sl) and h(l)(X,E(Sl)), respectively, when there is no ambiguity. Moreover, we denote
ES [ H(L)(X,S; θ) ] = ES [ H(L)(X,S; θ) ∣∣X] for convenience. Then,
EX [∥∥ESL+1[H(L+1)]− h(L+1)∥∥2] = EX
[∥∥∥ESL[EΓ(L+1)[fL+1(H(L) Γ(L+1))]− fL+1(h(L) γ(L+1))] +ESL [ fL+1(H (L) γ(L+1)) ] − fL+1(h(L) γ(L+1)) ∥∥∥ 2
] ≤ EX [∥∥∥ESL[EΓ(L+1)[fL+1(H(L) Γ(L+1))]− fL+1(h(L) γ(L+1))]∥∥∥ 2
] +EX [∥∥∥ESL[fL+1(H(L) γ(L+1))]− fL+1(h(L) γ(L+1))∥∥∥ 2
] From Eq. 2 and Jensen’s inequality, we have
EX [∥∥∥ESL[EΓ(L+1)[fL+1(H(L) Γ(L+1))]− fL+1(h(L) γ(L+1))]∥∥∥ 2 ] ≤ EH(L) [∥∥∥EΓ(L+1)[fL+1(H(L) Γ(L+1))]− fL+1(h(L) γ(L+1))∥∥∥ 2 ] ≤ δ
(3)
and
EX [∥∥∥ESL[fL+1(H(L) γ(L+1))]− fL+1(h(L) γ(L+1))∥∥∥ 2 ] = EX
[∥∥∥ESL[fL+1(H(L) γ(L+1))]− fL+1(ESL[H(L)] γ(L+1)) +fL+1(ESL [ H(L) ] γ(L+1))− fL+1(h(L) γ(L+1)) ∥∥∥ 2
] ≤ EX [∥∥∥ESL[fL+1(H(L) γ(L+1))]− fL+1(ESL[H(L)] γ(L+1))∥∥∥ 2
] +EX [∥∥∥fL+1(ESL[H(L)] γ(L+1))− fL+1(h(L) γ(L+1))∥∥∥ 2 ] (4)
Using Jensen’s inequality, property of operator norm and E [ Var[H(l)|X] ] ≤ σ2, we have
EX [∥∥∥ESL[fL+1(H(L) γ(L+1))]− fL+1(ESL[H(L)] γ(L+1))∥∥∥ 2 ] ≤ EH(L) [∥∥∥fL+1(H(L) γ(L+1))− fL+1(ESL[H(L)] γ(L+1))∥∥∥ 2
] ≤ BγEH(L)
[∥∥H(L) − ESL[H(L)]∥∥2] ≤ Bγ ( EH(L) [∥∥H(L) − ESL[H(L)]∥∥22]) 12 ≤ Bγσ (5)
From Eq. 1
EX [∥∥∥fL+1(ESL[H(L)] γ(L+1))− fL+1(h(L) γ(L+1))∥∥∥ 2 ] = BγEX
[∥∥ESL[H(L)]− h(L)∥∥2] ≤ Bγ∆L (6) Finally, to sum up with Eq. 3, Eq. 4, , Eq. 5, , Eq. 6, we have
EX [∥∥ESL+1[H(L+1)]− h(L+1)∥∥2] ≤ δ +Bγσ +Bγ∆L = (Bγ)Lδ + (δ +Bγσ) ( 1−(Bγ)L
1−Bγ
) = ∆L+1
C EXPECTATION-LINEARIZATION
C.1 PROOF OF THEOREM 4: UNIFORM DEVIATION BOUND
Before proving Theorem 4, we first define the notations.
Let Xn = {X1, . . . , Xn} be a set of n samples of input X . For a function space F : X → R, we use Radn(F , Xn) to denote the empirical Rademacher complexity of F ,
Radn(F , Xn) = Eσ [
sup f∈F ( 1 n n∑ i=1 σif(Xi) )]
and the Rademacher complexity is defined as Radn(F) = EXn [ Radn(F , Xn) ] In addition, we import the definition of dropout Rademacher complexity from Gao & Zhou (2014):
Rn(H, Xn, Sn) = Eσ [
sup h∈H
( 1 n n∑ i=1 σih(Xi, Si) )]
Rn(H) = EXn,Sn [ Radn(H, Xn, Sn) ]
where H : X × S → R is a function space defined on input space X and dropout variable space S. Rn(H, Xn, Sn) and Rn(H) are the empirical dropout Rademacher complexity and dropout Rademacher complexity, respectively. We further denoteRn(H, Xn) ∆ = ESn [ Radn(H, Xn, Sn) ] .
Now, we define the following function spaces: F = { f(x; θ) : f(x; θ) = ES [ H(L)(x, S; θ) ] , θ ∈ Θ } G = { g(x; θ) : g(x; θ) = h(L)(x,E[S]; θ), θ ∈ Θ
} H = { h(x, s; θ) : h(x, s; θ) = h(L)(x, s; θ), θ ∈ Θ
} Then, the function space of v(x) = f(x)− g(x) is V = {f(x)− g(x) : f ∈ F , g ∈ G}. Lemma 7.
Radn(F , Xn) ≤ Rn(H, Xn)
Proof. Rn(H, Xn) = ESn [ Radn(H, Xn, Sn) ] = ESn [ Eσ [ sup h∈H ( 1 n n∑ i=1 σih(Xi, Si) )]]
= Eσ [ ESn [ sup h∈H ( 1 n n∑ i=1 σih(Xi, Si) )]]
≥ Eσ [
sup h∈H
ESn [( 1 n n∑ i=1 σih(Xi, Si) )]]
= Eσ [ sup h∈H ( 1 n n∑ i=1 σiESi [ h(Xi, Si) ])] = Eσ [ sup h∈H ( 1 n n∑ i=1 σiESi [ H(L)(Xi, Si; θ) ])] = Radn(F , Xn)
From Lemma 7, we have Radn(F) ≤ Rn(H). Lemma 8.
Rn(H) ≤ αB LγL/2√ n
Radn(G) ≤ αB L
√ n
Proof. See Theorem 4 in Gao & Zhou (2014).
Now, we can prove Theorem 4.
Proof of Theorem 4
Proof. From Rademacher-based uniform bounds theorem, with probability ≥ 1− δ,
sup v∈V |∆− ∆̂| < 2Radn(V) + β
√ log(1/δ)
n
Since V = F − G, we have
Radn(V) = Radn(F − G) ≤ Radn(F) +Radn(G) ≤ αBL(γL/2 + 1)√
n
Then, finally, we have that with probability ≥ 1− δ,
sup θ∈Θ |∆− ∆̂| < 2αB L(γL/2 + 1)√ n + β
√ log(1/δ)
n
C.2 PROOF OF THEOREM 5: NON-UNIFORM BOUND OF MODEL ACCURACY
For convenience, we denote λ = {θ1, . . . , θL−1}. Then θ = {λ, η}, and MLE θ̂ = {λ̂, η̂} Lemma 9.
‖∇fL(·; η)T ‖op ≤ 2‖η‖2 (7)
Proof. denote
A = ∇fL(·; η)T = [ py(ηy − η)T ] ∣∣∣k y=1
where py = p(y|x, s; θ), η = E [ηY ] = k∑ y=1 pyηy .
For each v such that ‖v‖2 = 1,
‖Av‖22 = ∑ y∈Y ( py (ηy − η)T v )2 ≤ ∑ y∈Y ‖py (ηy − η) ‖22‖v‖22 = ∑ y∈Y ‖py (ηy − η) ‖22
≤ ∑ y∈Y py‖ηy − η‖22 ≤ ∑ y∈Y 2py ( ‖η‖22 + ∑ y′∈Y py′‖ηy′‖22 ) = 4
∑ y∈Y py‖ηy‖22 ≤ 4‖η‖22
So we have ‖A‖op ≤ 2‖η‖2.
Lemma 10. If parameter θ̃ = {λ̂, η} satisfies that ‖η‖2 ≤ δ4β , then V (D; θ̃) ≤ δ, where V (D; θ) is defined in Eq. (16).
Proof. Let SL = {Γ(1), . . . ,Γ(L)}, and let H(l) and h(l) be short for H(l)(X,Sl; θ̃) and h(l)(X,E(Sl); θ̃), respectively.
From lemma 9, we have ‖fL(x; η)− fL(y; η)‖2 ≤ 2‖η‖2‖x− y‖2. Then,∥∥ESL [HL]− hL∥∥2 = ∥∥ESL−1 [fL(H(L−1); η)]− fL(h(L−1); η)∥∥2 ≤ ESL−1
∥∥fL(H(L−1); η)− fL(h(L−1); η)∥∥2 ≤ 2‖η‖2 ∥∥H(L−1) − h(L−1)∥∥ 2 ≤ 4β‖η‖2 ≤ δ
Lemma 10 says that we can get θ satisfying the expectation-linearization constrain by explicitly scaling down η̂ while keeping λ̂.
In order to prove Theorem 5, we make the following assumptions:
• The dimension of h(L−1) is d, i.e. h(L−1) ∈ Rd. • Since ∀y ∈ Y, p(y|x; θ̂) > 0, we assume p(y|x; θ̂) ≥ 1/b, where b ≥ |Y| = k. • As in the body text, let p(y|x, s; θ̂) be nonuniform, and in particular let η̂Ty∗h (L−1)(x, s; λ̂)− η̂Ty h(L−1)(x, s; λ̂) > c‖η̂‖2,∀y 6= y∗.
For convenience, we denote ηTh(L−1)(x, s;λ) = ηTuy(x, s;λ), where uTy (x, s;λ) = (v T 0 , . . . , v T k ) and
vi = { h(L−1)(x, s;λ) if i = y 0 otherwise
To prove Theorem 5, we first prove the following lemmas.
Lemma 11. If p(y|x; θ̂) ≥ 1/b, then ∀α ∈ [0, 1], for parameter θ̃ = {λ̂, αη̂}, we have
p(y|x; θ̃) ≥ 1 b
Proof. We define
f(α) ∆ = (y|x, s; θ̃) = e αηTy h (L−1)(x,s;λ̂)∑
y′∈Y e αηT y′h (L−1)(x,s;λ̂)
=
( eη T y h (L−1)(x,s;λ̂) )α
∑ y′∈Y ( e ηT y′h (L−1)(x,s;λ̂) )α
Since Y = {1, . . . , k}, for fixed x ∈ X , s ∈ S, log f(α) is a concave function w.r.t α. Since b ≥ k, we have
log f(α) ≥ (1− α) log f(0) + α log f(1) ≥ − log b
So we have ∀x, s, p(y|x, s; θ̃) ≥ 1/b. Then
p(y|x; θ̃) = ES [ p(y|x, S; θ̂) ] ≥ 1 b
Lemma 12. if y is not the majority class, i.e. y 6= y∗, then for parameter θ̃ = {λ̂, αη̂}
p(y|x, s, θ̃) ≤ e−cα‖η̂‖2
Proof.
p(y|x, s, θ̃) = e αη̂Tuy∑
y′∈Y eαη̂
Tuy′ ≤ e
αη̂Tuy
eαη̂ Tuy∗
≤ e−cα‖η̂‖2
Lemma 13. For a fixed x and s, the absolute value of the entry of the vector under the parameter θ̃ = {λ̂, αη̂}:
|p(y|x, s; θ̃)(uy − EY [uY ])|i ≤ β(k − 1)e−cα‖η̂‖2
Proof. Suppose y is the majority class of p(y|x, s; θ̃). Then,
uy − Ey[uY ] = (vy′)ky′=1
where
vy = { (1− p(y|x, s; θ̃)h(L−1) if y = y∗ −p(y|x, s; θ̃)h(L−1) otherwise
From Lemma 12, we have
|p(y|x, s; θ̃)(uy − EY [uY ])|i ≤ |(uy − EY [uY ])|i ≤ β(k − 1)e−cα‖η̂‖2
Now, we suppose y is not the majority class of p(y|x, s; θ̃). Then,
|p(y|x, s; θ̃)(uy − EY [uY ])|i ≤ p(y|x, s; θ̃)β ≤ βe−cα‖η̂‖2
Overall, the lemma follows.
Lemma 14. We denote the matrix
A ∆ = ES [ p(y|x,s;θ̃) p(y|x;θ̃) (uy − EY [uY ])(uy − EY [uY ]) T ]
−ES [ p(y|x,s;θ̃) p(y|x;θ̃) (uy − EY [uY ]) ] ES [ p(y|x,s;θ̃) p(y|x;θ̃) (uy − EY [uY ]) ]T Then the absolute value of the entry of A under the parameter θ̃ = {λ̂, αη̂}:
|Aij | ≤ 2b(k − 1)β2e−cα‖η̂‖2
Proof. From Lemma 11, we have p(y|x; θ̃) ≥ 1/b. Additionally, the absolute value of the entry of uy − EY [uY ] is bounded by β. We have for each i∣∣∣∣∣ES [ p(y|x, s; θ̃) p(y|x; θ̃) (uy − EY [uY ]) ]∣∣∣∣∣ i ≤ ES [ p(y|x, s; θ̃) p(y|x; θ̃) β ] = β
Then from Lemma 13 |Aij | ≤ 2b(k − 1)β2e−cα‖η̂‖2
Lemma 15. We denote the matrix
B ∆ = ES [ p(y|x, s; θ̃) p(y|x; θ̃) ( EY [ uY u T Y ] − EY [uY ]EY [uY ]T )]
Then the absolute value of the entry of B under the parameter θ̃ = {λ̂, αη̂}:
|Bij | ≤ 2(k − 1)β2e−cα‖η̂‖2
Proof. We only need to prove that for fixed x and s, for each i, j:∣∣EY [uY uTY ]− EY [uY ]EY [uY ]T ∣∣ij ≤ 2(k − 1)β2e−cα‖η̂‖2 Since ∣∣EY [uY uTY ]− EY [uY ]EY [uY ]T ∣∣ij = |CovY [(uY )i, (uY )j ]| ≤ β2 k∑ y=1 p(y|x, s; θ̃)− p(y|x, s; θ̃)2
Suppose y is the majority class. Then from Lemma 12,
p(y|x, s; θ̃)− p(y|x, s; θ̃)2 ≤ 1− p(y|x, s; θ̃) ≤ (k − 1)e−cα‖η̂‖2
If y is not the majority class. Then,
p(y|x, s; θ̃)− p(y|x, s; θ̃)2 ≤ p(y|x, s; θ̃) ≤ e−cα‖η̂‖2
So we have k∑ y=1 p(y|x, s; θ̃)− p(y|x, s; θ̃)2 ≤ 2(k − 1)e−cα‖η̂‖2
The lemma follows.
Lemma 16. Under the parameter θ̃ = {λ̂, αη̂}, the largest eigenvalue of the matrix
1
n n∑ i=1 (A(xi, yi)−B(xi, yi)) (8)
is at most 2dk(k − 1)(b+ 1)β2e−cα‖η̂‖2
Proof. From Lemma 14 and Lemma 15, each entry of the matrix in (8) is at most 2(k − 1)(b + 1)β2e−cα‖η̂‖2 . Thus, by Gershgorin’s theorem, the maximum eigenvalue of the matrix in (8) is at most 2dk(k − 1)(b+ 1)β2e−cα‖η̂‖2 .
Now, we can prove Theorem 5 by constructing a scaled version of θ̂ that satisfies the expectationlinearization constraint.
Proof of Theorem 5
Proof. Consider the likelihood evaluated at θ̃ = {λ̂, αη̂}, where α = δ4β‖η̂‖2 . If α > 1, then ‖η‖2 > δ4β . We know the MLE θ̂ already satisfies the expectation-linearization constraint. So we can assume that 0 ≤ α ≤ 1, and we know that θ̃ satisfies V (D; θ̃) ≤ δ. Then,
∆l(θ̂, θ̂δ) ≤ ∆l(θ̂, θ̃) = 1
n (l(D; θ̂)− l(D; θ̃)) = g(λ̂, η̂)− g(λ̂, αη̂)
where g(λ, η) = 1n l(D; (λ, η)). Taking the second-order Taylor expansion about η, we have
g(λ̂, αη̂) = g(λ̂, η̂) +∇Tη g(λ̂, η̂)(αη̂ − η̂) + (αη̂ − η̂)T∇2ηg(λ̂, η̂)(αη̂ − η̂)
Since θ̂ is the MLE, the first-order term ∇Tη g(λ̂, η̂)(αη̂ − η̂) = 0. The Hessian in the second-order term is just Eq.(8). Thus, from Lemma 16 we have
g(λ̂, αη̂) ≤ g(λ̂, η̂)− (1− α)2‖η̂‖222dk(k − 1)(b+ 1)β2e−cα‖η̂‖2 = g(λ̂, η̂)− 2dk(k − 1)(b+ 1)β2 ( ‖η̂‖2 − δ4β )2 e−cδ/4β
= g(λ̂, η̂)− c1β2 ( ‖η̂‖2 − δ4β )2 e−c2δ/4β
with setting c1 = 2dk(k − 1)(b+ 1) and c2 = c. Then the theorem follows.
C.3 PROOF OF THEOREM 6: UNIFORM BOUND OF MODEL ACCURACY
In the following, we denote θ̃ = {λ̂, αη̂}.
Lemma 17. For each y ∈ Y , if p(y|x, s; θ̂) ≥ 1/k, then ∀α ∈ [0, 1]
p(y|x, s; θ̃) ≥ 1 k
Proof. This lemma can be regarded as a corollary of Lemma 11.
Lemma 18. For a fixed x and s, we denote eη̂ T y h (L−1)(x,s;λ̂) = wy . Then we have
p(y|x, s, θ̃) = e αη̂Ty h (L−1)(x,s;λ̂)∑ y′∈Y e αη̂T y′h (L−1)(x,s;λ̂) = (wy) α∑ y′∈Y (wy′)α
Additionally, we denote gs(α) = ∑ y′∈Y p(y′|x, s; θ̃) logwy′ − logwy . We assume gs(0) ≥ 0. Then we have ∀α ≥ 0 gs(α) ≥ 0
Proof.
∂gs(α) ∂α = ∑ y′∈Y logwy′ ∂p(y′|x, s; θ̃) ∂α = VarY [logwY |X − x, S = s] ≥ 0
So gs(α) is non-decreasing. Since gs(0) ≥ 0, we have gs(α) ≥ 0 when α ≥ 0.
From above lemma, we have for each training instance (xi, yi) ∈ D, and ∀α ∈ [0, 1],
EY [ log p(Y |xi, s; θ̃) ] ≥ log p(yi|xi, s; θ̃) (9)
For convenience, we define m(s, y) = log p(y|x, s; θ̃)− EY [ log p(Y |x, s; θ̃) ]
Lemma 19. If y satisfies Lemma 17 and gs(α) ≥ 0, then
VarY [m(s, Y )] ≥ m(s, y)2
Proof. First we have m(s, y) = log p(y|x, s; θ̃)− log 1/k −KL ( p(·|x, s; θ̃)|Unif(Y) ) ≤ 0
So we have
(VarY [m(s, Y )]) 1/2 = √ EY [( log p(Y |x, s; θ̃)− EY [ log p(Y |x, s; θ̃) ])2] ≥ EY
[∣∣∣log p(Y |x, s; θ̃)− EY [log p(Y |x, s; θ̃)]∣∣∣] = EY
[∣∣∣KL(p(·|x, s; θ̃)|Unif(Y))+ log 1/k − log p(Y |x, s; θ̃)∣∣∣] = EY [ KL ( p(·|x, s; θ̃)|Unif(Y) ) + ∣∣∣log 1/k − log p(Y |x, s; θ̃)∣∣∣]
≥ KL ( p(·|x, s; θ̃)|Unif(Y) ) + EY [ log p(Y |x, s; θ̃)− log 1/k ] = 2KL ( p(·|x, s; θ̃)|Unif(Y)
) As KL ( p(·|x, s; θ̃)|Unif(Y) ) ≥ 0 and log p(y|x, s; θ̃) ≥ log 1/k. So we have 2KL ( p(·|x, s; θ̃)|Unif(Y) ) ≥ KL ( p(·|x, s; θ̃)|Unif(Y) ) +log 1/k−log p(y|x, s; θ̃) = −m(s, y)
Then the lemma follows.
From Lemma 19 and Eq. (9), we have for each training instance (xi, yi) ∈ D, and ∀α ∈ [0, 1],
VarY [m(s, Y )] ≥ m(s, yi)2 (10)
Lemma 20. For each training instance (xi, yi) ∈ D, and ∀α ∈ [0, 1], we have
log p(yi|xi; {λ̂, αη̂}) ≥ (1− α) log p(yi|xi; {λ̂, 0}) + α log p(yi|xi; {λ̂, η̂})
Proof. We define
f(α) = log p(yi|xi; {λ̂, αη̂})− (1− α) log p(yi|xi; {λ̂, 0})− α log p(yi|xi; {λ̂, η̂})
Because f(0) = f(1) = 0, we only need to prove that f(α) is concave on [0, 1]. We have
∇2f(α) = −ES|Y=yi [VarY [m(S, Y )]] + VarS|Y=yi [m(S, yi)]
where S|Y = yi is under the probability distribution p(s|Y = yi, xi; θ̃) = p(yi|xi,S;θ̃)p(s)p(yi|xi;θ̃) From Eq. (10), we have
ES|Y=yi [VarY [m(S, Y )]] ≥ ES|Y=yi [ m(S, yi) 2 ] ≥ VarS|Y=yi [m(S, yi)]
So we have ∇2f(α) ≤ 0. The lemma follows.
Now, we can prove Theorem 6 by using the same construction of an expectation-linearizing parameter as in Theorem 5.
Proof of Theorem 6
Proof. Consider the same parameter θ̃ = {λ̂, αη̂}, where α = δ4β‖η̂‖2 ≤ 1. we know that θ̃ satisfies V (D; θ̃) ≤ δ. Then,
∆l(θ̂, θ̂δ) ≤ ∆l(θ̂, θ̃) = 1
n (l(D; θ̂)− l(D; θ̃))
From Lemma 20 we have:
l(D; θ̃) = l(D; {λ̂, αη̂}) ≥ (1− α)l(D; {λ̂, 0}) + αl(D; {λ̂, η̂})
So ∆l(θ̂, θ̂δ) ≤ (1− α) 1n ( l(D; θ̂)− l(D; {λ̂, 0}) ) = (1− α) 1n n∑ i=1 log p(yi|xi; θ̂)− log Unif(Y)
(1− α)E [KL (p(·|X; θ)‖Unif(Y))] ≤ ( 1− δ4β‖η̂‖2 ) E [KL (p(·|X; θ)‖Unif(Y))]
D DETAILED DESCRIPTION OF EXPERIMENTS
D.1 NEURAL NETWORK ARCHITECTURES
MNIST For MNIST, we train 6 different fully-connected (dense) neural networks with 2 or 3 layers (see Table 1). For all architectures, we used dropout rate p = 0.5 for all hidden layers and p = 0.2 for the input layer.
CIFAR-10 and CIFAR-100 For the two CIFAR datasets, we used the same architecture in Srivastava et al. (2014) — three convolutional layers followed by two fully-connected hidden layers. The convolutional layers have 96, 128, 265 filters respectively, with a 5× 5 receptive field applied with a stride of 1. Each convolutional layer is followed by a max pooling layer pools 3× 3 regions at strides of 2. The fully-connected layers have 2048 units each. All units use the rectified linear activation function. Dropout was applied to all the layers with dropout rate p = (0.1, 0.25, 0.25, 0.5, 0.5, 0.5) for the layers going from input to convolutional layers to fully-connected layers.
D.2 NEURAL NETWORK TRAINING
Neural network training in all the experiments is performed with mini-batch stochastic gradient descent (SGD) with momentum. We choose an initial learning rate of η0, and the learning rate is updated on each epoch of training as ηt = η0/(1 + ρt), where ρ is the decay rate and t is the number of epoch completed. We run each experiment with 2,000 epochs and choose the parameters achieving the best performance on validation sets.
Table 3 summarizes the chosen hyper-parameters for all experiments. Most of the hyper-parameters are chosen from Srivastava et al. (2014). But for some experiments, we cannot reproduce the performance reported in Srivastava et al. (2014) (We guess one of the possible reasons is that we used different library for implementation.). For these experiments, we tune the hyper-parameters on the validation sets by random search. Due to time constrains it is infeasible to do a random search across the full hyper-parameter space. Thus, we try to use as many hyper-parameters reported in Srivastava et al. (2014) as possible.
D.3 EFFECT OF EXPECTATION-LINEARIZATION RATE λ
Table 4 illustrates the detailed results of the experiments on the effect of λ. For MNIST, it lists the error rates under different λ values for six different network architectures. For two datasets of CIFAR, it gives the error rates under different λ values, among with the empirical expectation-linearization risk ∆̂.

summary
The paper explains dropout with a latent variable model where the dropout variable (0 or 1 depending on which units should be dropped) is not observed and is accordingly marginalised. Maximum likelihood under this model is not tractable but standard dropout then corresponds to a simple Monte Carlo approximation of ML for this model.
The paper then introduces a theoretical framework for analysing the discrepancy (called inference gap) between the model at training (model ensemble, or here the latent variable model), and the model at testing (where usually what should be an expectation over the activations over many models becomes the activation of one model with averaged weights).
This framework introduces several notions (e.g. expectation linearity) which allow the study of which transition functions (and more generally layers) can have a small inference gap. Theorem 3 gives a bound on the inference gap.
Finally a new regularisation term is introduced to account for minimisation of the inference gap during learning.
Experiments are performed on MNIST, CIFAR-10 and CIFAR-100 and show that the method has the potential to perform better than standard dropout and at the level of Monte Carlo Dropout (the standard method to compute the real dropout outputs consistently with the training assumption of an ensemble, of course quite expensive computationally)
The study gives a very interesting theoretical model for dropout as a latent variable model where standard dropout is then a monte carlo approximation. This is very probably widely applicable to further studies of dropout.
The framework for the study of the inference gap is interesting although maybe somewhat less widely applicable.
The proposed model is convincing although 1. it is tested on simple datasets 2. the gains are relatively small and 3. there is an increased computational cost during training because a new hyper-parameter is introduced.
p6 line 8 typo: expecatation