We tackle the issue of finding a good policy when the number of policy updates is limited. This is done by approximating the expected policy reward as a sequence of concave lower bounds which can be efficiently maximized, drastically reducing the number of policy updates required to achieve good performance. We also extend existing methods to negative rewards, enabling the use of control variates.
We tackle the issue of finding a good policy when the number of policy updates is limited. This is done by approximating the expected policy reward as a sequence of concave lower bounds which can be efficiently maximized, drastically reducing the number of policy updates required to achieve good performance. We also extend existing methods to negative rewards, enabling the use of control variates.
1 INTRODUCTION
Recently, reinforcement learning has seen a surge in popularity, in part because of its successes in playing Atari games (Mnih et al., 2013) and Go (Silver et al., 2016). Due to its ability to act in settings where the actions taken influence the environment and, more generally, the input distribution of examples, reinforcement learning is now used in other domains, such as online advertising.
The goal is to learn a good policy, i.e. a good mapping from states to actions, which will maximize the final score, in the case of Atari games, the probability of winning, in Go, or the number of sales, in online advertising. We have at our disposal logs of past events, where we know the states we were in, the actions we took and the resulting rewards we obtained. In this paper, we shall focus on how to efficiently use these logs to obtain a good policy. In some cases, in addition to (state, action, reward) triplets, we have access to a teacher which provides the optimal, or at least a good, action for a given state. The use of such a teacher to find a good initial policy is outside the scope of this paper.
There are many ways to learn good policies using past data. The two most popular are Q-learning and direct policy optimization. In Q-learning (Sutton & Barto, 1998), we are trying to learn a mapping from a (state, action) pair to the reward. Given this mapping and a state, we can then find the action which leads to the maximal predicted reward. This method has been very successful, especially when the action space is small, since we need to test all the actions, and the reward somewhat predictable, since taking the maximum is unstable and a small error can lead to suboptimal actions.
Direct policy optimization, rather than trying to estimate the value of a (state, action) pair, directly parameterizes a policy, i.e. a conditional distribution over actions given the current state. More precisely, and using the notation from Kober (2014), we wish to maximize the expected return of a policy p(·|θ) with parameters θ, i.e.
J(θ) = ∫ T p(τ |θ)R(τ) dτ , (1.1)
where T is the set of all paths τ and R(τ) is the reward associated with path τ . A rollout τ = [s1:T+1, a1:T ] is a series of states s1:T+1 = [s1, . . . , sT+1] and actions a1:T = [a1, . . . , aT ]. p(τ |θ) is the probability of rollout τ when using a policy with parameters θ and R(τ) is the aggregated return of τ . If we make the Markov assumption that a state only depends on the
previous state and the action chosen, we have p(τ |θ) = p(s1) T∏ t=1 p(st+1|st, at)π(at|st, t, θ) where p(st+1|st, at) is the next state distribution and is independent of our policy. The action space can be discrete or continuous.
Without the ability to exactly compute J a, we must resort to sampling to get an estimate of both J and its gradient. These samples can come from p(·|θ) or from another distribution. In the latter case, we need to use importance sampling to keep our estimator unbiasedb. Whether they use importance sampling or not, all methods which directly optimize the policy rely on iterative procedures. First, rollouts are performed under a policy to gather samples. Then, these samples are used to update the policy, which is in turn used to gather new samples, until convergence of the policy. These methods continuously gather new samples, mostly because the updates to the policy are more reliable when they are based on fresh samples. However, in a production environment, as we will see in Sec. 2, we will typically release a new policy to many users at once, gathering millions or billions of samples, but the delay between two updates of a policy is of the order of hours or even days. Thus, there is a strong need for policy learning techniques which can achieve a good performance with a limited number of policy updates. There is an analogous issue in robotics where each new rollout induces wear and tear on the robot, making such a method which limits the number of rollouts desirable.
In this paper, we shall thus present a method dedicated to achieving high performance while limiting the number of different policies under which samples are gathered. Sec. 3 reviews the relevant literature. Sec. 4 presents a first version of our algorithm, proving its theoretical efficiency and providing a common framework to several existing techniques. Then, Sec. 5 shows how the positiveness assumption for the rewards can slow down learning and proposes an improvement which circumvents the issue. Sec. 6 shows results of the proposed algorithm on both a synthetic dataset and a real-life, large scale dataset in online advertising. Finally, Sec. 7.3 reflects on the current state and the future venues of research.
2 DISPLAY ADVERTISING
Retargeting is a type of advertising where ads are displayed to users who have already expressed interest in one or multiple products, generally by browsing on merchants’ websites. Since the information used to know which ad to display are not related to a query, like in search advertising, these ads can be displayed on any website, for instance news sites or personal pages. More precisely, every time a user lands on such a website, the website contacts an ad-exchange platform which runs a real-time auction. The highest bidder gets the right to display an ad for this particular user and pays a price depending on many factors, including the opponents’ bids. If the user then clicks on the ad, the retargeter is paid by the merchant whose ad was displayed. To maximize its revenue, the retargeter must thus only display ads leading to a click, and do so at the lowest possible price. Historically, these auctions were second-price, which means that the price paid by the highest bidder was the second highest bid. From a bidder perspective, the optimal strategy was straightforward as the optimal bid was the expected gain of displaying an ad. However, ad-exchanges have recently moved to other types of auctions, where the optimal strategy depends on the (unknown) bids of the other bidders. Worse, the exact type of auction is unknown to the bidders who only know the price they pay when they win the auction.
The bidding problem thus fits nicely in the reinforcement learning framework where the state is the set of information known about the user and the current website, the action is the bid and the reward is the payment (if there is a click) minus the cost of displaying the ad. As the reward depends mostly on whether there is a click or not, an event which is highly unpredictable, techniques such as doubly robust estimation (Dudı́k et al., 2011) or based on carefully crafted Q-functions (Lillicrap et al., 2015; Schulman et al., 2015b; Gu et al., 2016) are unlikely to yield significant improvements.
There is another major difference with other reinforcement learning works. For quality control, a new policy can only be put in production every few hours or even days. Since large advertising companies display several billion ads each day, each new policy is used to gather about a billion samples. We are thus in a very specific setting where the number of samples is very large but the number of policies with which samples are gathered is limited.
We will now review the existing literature and show how no existing work addresses the constraints we face. We will then present our solution which is both simple and leads to good policies. To
aComputing J would require visiting every possible τ at least once, which is impossible, even for moderately long rollouts.
bUsing a biased estimator can be useful but this is outside the scope of this paper.
show its efficiency, we report results on both the Cartpole problem, a synthetic problem common in reinforcement learning, and a real-world example.
3 RELATED WORK
We review here some of the most common techniques in reinforcement learning, limiting ourselves to those who try to directly optimize the policy.
The first such method is REINFORCE (Williams, 1992), which performs a single gradient step between two rollouts of the policy. This method has multiple issues. First, one has to do rollouts after each update, ultimately resulting in many rollouts before reaching a good solution. This is further emphasized with the potential poor quality of the gradient update which is not insensitive to a reparametrization of the parameter space. Finally, as with any stochastic method, the choice of the stepsize has a strong influence of the convergence speed. Each of these problems has been treated in separate works. The need to perform rollouts after each update was alleviated by using importance sampling policy gradient (Swaminathan & Joachims, 2015). The update direction can be improved by using natural gradient (Amari, 1998; Kakade, 2001) and doing a line search helps in choosing a correct stepsize (Jie & Abbeel, 2010). These improvements can be computationally expensive and the additional hyperparameters make them less suited to a production environment where simplicity and robustness are key.
Another line of research used concave lower bounds on J(θ), which could then be optimized using off-the-shelf classifiers such as L-BFGS (Liu & Nocedal, 1989). Examples of such methods are PoWER (Kober & Peters, 2009) and Natural actor-critic (Peters & Schaal, 2008). These bounds were obtained using an analogy with EM (Dempster et al., 1977; Minka) which required the rewards R(τ) to be positive. In settings where multiple policies can achieve high accuracy, Neumann (2011) proposed another EM-based method which focuses on one of these policies rather than trying to loosely cover all of them, at the expense of a larger computational cost.
We will show how these works can be extended to better optimize the policy between two updates. We will then show how the positiveness requirement hurts the optimization, then propose an extension which allows us to use any reward.
Since, in practice, we do not have access to the full distribution but rather to a set of N samples, we shall optimize a Monte-Carlo estimate of J :
Ĵ(θ) = 1
N ∑ i R(τi) p(τi|θ) p(τi|θ0) , τi ∼ p(τ |θ0) , (3.1)
where p(τ |θ0) is the probability of rollout τ under the distribution used to generate samples. This is the standard importance sampling trick commonly used in policy gradient (Sutton et al., 1999).
4 CONCAVE APPROXIMATION OF THE EXPECTED LOSS
In this section, we assume that all returns are nonnegativec. Due to this nonnegativity, the nonconcavity in J stems from the nonconcavity of each p(τ |θ). However, if p(τ |θ) belongs to the exponential family (Wainwright & Jordan, 2008), then it is a log-concave function of its parameters and log p(τ |θ) is a concave function of θ. This suggests the following lower bound:
Lemma 1. Let
pq(τ |θ) = q(τ) ( 1 + log
p(τ |θ) q(τ)
) (4.1)
with q such that q(τ) 6= 0 when p(τ |θ) 6= 0. Then we have pq(τ |θ) ≤ p(τ |θ).
cOr at least bounded below, in which case they need to be adequately shifted.
Proof.
p(τ |θ) = q(τ)p(τ |θ) q(τ)
≥ q(τ) ( 1 + log
p(τ |θ) q(τ) ) = pq(τ |θ) .
The second line stems from the inequality x ≥ 1 + log x.
Lemma 1 shows that, regardless of the function q chosen, pq(τ |θ) is a lower bound of p(τ |θ) for any value of θ. Thus, provided that p(τ |θ) belongs to the exponential family, we have obtained a log-concave lower bound. Lemma 1, however, does not guarantee the quality of that lower bound. This is addressed by the following lemma:
Lemma 2. If there is a ν such that q(τ) = p(τ |ν), we have
pq(τ |ν) = p(τ |ν) , ∂pq(τ |θ) ∂θ ∣∣∣∣ θ=ν = ∂p(τ |θ) ∂θ ∣∣∣∣ θ=ν .
Proof. pq(τ |ν) = p(τ |ν) is immediate when setting θ = ν in Eq. 4.1. Deriving pq(τ |θ) with respect to θ yields
∂pq(τ |θ) ∂θ = p(τ |ν)∂ log p(τ |θ) ∂θ
= p(τ |ν) p(τ |θ) ∂p(τ |θ) ∂θ .
Taking θ = ν on both sides yields ∂pq(τ |θ)∂θ ∣∣∣∣ θ=ν = ∂p(τ |θ)∂θ ∣∣∣∣ θ=ν .
To simplify further notations, we will write directly pν(τ |θ) = p(τ |ν) ( 1 + log
p(τ |θ) p(τ |ν)
) (4.2)
to explicitly show the dependency of the bound on the parameter ν.
The following result is a direct consequence of these two lemmas:
Lemma 3. (lower bound of the expected reward estimator): Let
Ĵν(θ) = 1
N ∑ i R(τi) p(τi|ν) p(τi|θ0) ( 1 + log p(τi|θ) p(τi|ν) ) . (4.3)
Then we have Ĵν(θ) ≤ Ĵ(θ) ∀θ , Ĵν(ν) = Ĵ(ν) , ∂Ĵν(θ)
∂θ ∣∣∣∣ θ=ν = ∂Ĵ(θ) ∂θ ∣∣∣∣ θ=ν . Further, if
p(τ |·) is a log-concave function, then Ĵν is concave for any ν.
Proof. Since each R(τi) is nonnegative, so is the ratio R(τi) p(τi|θ0) . The sum of lower bounds being a lower bound, this concludes the proof.
It is now worth going into more detail on the three parameters of Eq. 4.3:
• θ is the current value of the parameter we are trying to optimize over; • θ0 is the parameter value used to gather samples; • ν is the parameter used to create the lower bound. Any value of ν is valid.
There are two special cases of this bound. First, when ν = θ = θ0, this bound becomes an equality and we recover the policy gradient of Williams (1992). However, this equality only holds for the first update of θ. Second, a more interesting case is ν = θ0 6= θ. In this case, Eq. 4.3 simplifies and we get
Ĵθ0(θ) = 1
N ∑ i R(τi) ( 1 + log p(τi|θ) p(τi|θ0) ) . (4.4)
This bound is used by multiple authors (Dayan & Hinton, 1997; Peters & Schaal, 2007; 2008; Kober, 2014; Schulman et al., 2015a) and has the attractive property that it is tight at the beginning of the optimization since we have θ = θ0. When we optimize this bound without ever changing the value of ν, we end up with exactly the PoWER algorithm. However, as we optimize θ, this bound becomes looser and it might be useful to change the value of ν.
This suggest an iterative scheme where, after the optimization of Eq. 4.4 has ended in θ = θ1, we recompute the bound of Eq. 4.3 with ν = θ1. This yields an iterative version of the PoWER algorithm as described in Algorithm 1.
The data: Rewards R(τi), probabilities p(τi|θ0), initial parameters θ0 The result: Final parameters θT for t = 1 to T do
θt = argmax θ Ĵθt−1(θ)
= argmax θ ∑ i R(τi) p(τi|θt−1) p(τi|θ0) ( 1 + log p(τi|θ) p(τi|θt−1) ) end
Algorithm 1: Iterative PoWER
We recall that PoWER is equivalent to Algorithm 1 but with T = 1. As we shall see in the experiments, larger values of T lead to vast improvements.
One can also see that Algorithm 1 performs the same optimizations as the PoWER algorithm where new samples would be gathered at each iteration, with the difference that importance sampling is used rather than true samples. Thus, in the spirit of off-policy learning, we have replaced extra sampling with importance sampling. When, and only when, variance starts to be too high, can we sample from the new distribution.
5 CONVEX UPPER BOUND
Lemma 3 requires positive returns. This has two undesirable consequences. First, this can lead to very slow convergence. One can see this by creating a setting where all rollouts lead to a positive return except for one which leads to a return of −β < 0. To maintain the positivity of the returns, we need to shift all the returns by β which does not change the optimal policy using the following
transformation: J(θ) = ∫ T p(τ |θ)R(τ) dτ = ∫ T p(τ |θ)(R(τ) + β) dτ − β. We may now apply lemma 3 and optimize the following lower bound:
Jβν (θ) = ∫ T p(τ |ν) ( 1 + log p(τ |θ) p(τ |ν) ) (R(τ) + β) dτ − β .
Without the rollout with a negative return, the returns would not need to be shifted by β and we could have optimized J0ν (θ) instead. The difference between the two is equal to J β ν (θ) − J0ν (θ) = −βKL(p(τ |ν)||p(τ |θ)) where KL is the Kullback-Leibler divergence, which encourages p(τ |θ) to be close to p(τ |ν). Hence, one rollout with a negative return would slow down our optimization with a regularization term proportional to that return. A simple heuristic would be to discard such rollouts but we would lose all guarantees about the improvement of the expected return.
Further, the positivity of the returns precludes the use of control variates which are especially useful when the shifted rewards are approximately centered on 0. Thus, it prevents us from benefiting of all the existing techniques based around these control variates which would help reducing the variance.
The positivity assumption is required since we multiply our lower bound with the returns. If, instead, we have convex upper bounds of p(τ |θ), then this would provide us with a concave lower bound whenever it is associated with a negative return. Lemma 4 provides such a bound. Lemma 4. Let
uν(τ |θ) = p(τ |ν) exp [ (θ − ν)> ∂ log p(τ |θ)
∂θ
∣∣∣∣ θ=ν ] , (5.1)
where p(τ |θ) is a log-concave function of θ. Then uv(τ |θ) is a convex function of θ and we have:
uν(τ |ν) = p(τ |ν) , ∂uν(τ |θ)
∂θ ∣∣∣∣ θ=ν = ∂p(τ |θ) ∂θ ∣∣∣∣ θ=ν
, uν(τ |θ) ≥ p(τ |θ) ∀θ .
Proof. Both equalities can be verified by setting θ = ν. Since uν is the exponential of a linear function in its argument, it is convex. Finally, using the concavity of log p, we have:
log p(τ |θ) ≤ log p(τ |ν) + (θ − ν)> ∂ log p(τ |θ) ∂θ ∣∣∣∣ θ=ν
p(τ |θ) ≤ p(τ |ν) exp (θ−ν)> ∂ log p(τ|θ)∂θ ∣∣∣∣ θ=ν .
This concludes the proof.
We may now combine the two bounds to get the following lower bound on Ĵ(θ) without any constraint on the rewards:
Ĵ(θ) ≥ 1 N ∑ i R(τi) p(τi|ν) p(τi|θ0) zi(θ) (5.2)
zi(θ) = 1R(τi)≥0
( 1 + log
p(τi|θ) p(τi|ν)
) + 1R(τi)<0 exp [ (θ − ν)> ∂ log p(τ |θ)
∂θ
∣∣∣∣ θ=ν ] .
Further, when p(τ |θ) is log-concave in θ, then zi(θ) is concave in θ. This allows us to deal with positive and negative rewards, which means that the choice of control variate is now free, which can help both in reducing the variance and improving the lower bound and thus the convergence.
6 EXPERIMENTS
To demonstrate the effectiveness of Iterative PoWER, we provide results obtained on the Cartpole benchmark using the Gym d toolkit. We also show experiments on real online advertising data where we try to maximize advertisers’ revenue while keeping our costs constant.
6.1 CARTPOLE BENCHMARK
To show how we can achieve good performance with limited rollouts, we ran our experiments on the Cartpole benchmark, which is solved by the PoWER method. We used a stochastic linear policy with a 4-dimensional state (positions and velocities of the cart and the pole) where, at each time step, the cart moves right with probability σ(s>θ) where s is the vector representation of the state. Each experiment was 250 rollouts, in 10 batches of 25 rollouts, of length 400. In between each batch, we retrained the parameters of the model. The average performance over 350 repetitions was computed every batch, varying the number T of iterations of PoWER e. We capped the importance weights at 20.
dhttps://gym.openai.com/ eEach iteration of PoWER consisted in 5 steps of Newton method
Figure 3: Dashed blue: improvement in expected merchant value as a function of the number of iterations (arbitrary linear scale). The optimization is very slow at the beginning and the improvement is close to linear for the first 50 iterations. Solid green: Relative change of the expected cost as a function of the number of iterations. The expected spend increased up to 0.0086% but reached 0.0001% when we stopped the optimization.
We also evaluated the impact of using control variates on the convergence. To that effect, at each iteration, we computed the control variate minimizing the variance of the total estimator. We then used as control variate fractions of that value. For instance, cv = 0.5 means that the control variate used was half of the “optimal” control variate found by minimizing the variance of the estimator. The results can be seen in Fig. 2 and Fig. 1. We see that doing several iterations of PoWER leads to an increase in performance, up to a point where the variance is too high.
We can also see that, without using control variates (cv = 0), it is not beneficial to do too several iterations. The stagnation in performance for cv = 0.25 and cv = 0.5 might be due to the poorer quality of the upper bound compared to the lower bound. Thus, to get the full benefit of the control variate, we need to use larger values, such as cv = 0.99, which yields the highest performance for any value of T .
6.2 REAL WORLD DATA - ONLINE ADVERTISING
We also tested Iterative PoWER using real data gathered from a production system. In this system, every time a user lands on a webpage, an auction is run to determine who gets to display an ad to that user. Each participant to the auction chooses in real-time how much to bid for the right to display that ad. If the ad is clicked and then the user buys an item on the merchant’s website, a reward equal to the value of the item is observed. Otherwise, a reward of 0 is observed. The cost of displaying an ad depends on the bid in a way unknown to us as we do not have access to the type of auction nor to the bids of the other participants.
We gathered data for over 1.3 billion displays over a period of 2 weeks in April 2016. This data was comprised of information available at the time of the bid, for instance the history of the user, the current URL, or the size of the ad to display. The aggregation of all this data represented the states. When we won the auction, we also logged our bid, which are our actions, the cost of the display and the value generated if there was a sale, both of which are used for the reward.
Rather than learning a full bidding strategy mapping from states to a bid, we used our production system as baseline and learnt a small modifier to account for the non-truthfulness of the auctions. Thus, only a few thousand parameters were learnt, a small number compared to the number of training samples. This allowed us to run many iterations of PoWER without fear of high variance.
Since our aim was to maximize the value generated, this could lead to the undesirable solution where all ads are bought regardless of their price. Thus, we included the constraint that the total cost of buying the ads had to remain constant. The details of iPoWER with added constraints are in section 7.1
Fig. 3 shows the relative improvement in advertiser value generated as a function of the number T of iterations. We can readily see that the final improvement obtained by Iterative Power is far greater than that obtained after one iteration of PoWER (about 60 times greater). The green curve shows the change in total cost. Since we also used a lower bound for the constraint, it is initially not satisfied, but running the algorithm to convergence leads to a solution in the feasible set. The final improvement represents an increase of the net gain by several percentage points.
7 EXTENSIONS AND CONCLUSION
7.1 APPLICATION TO CONSTRAINED OPTIMIZATION
There are cases where one wishes to maximize the expected reward while satisfying a constraint on another quantity. In online advertising, for instance, it is interesting to maximize the number of clicks (or sales) while keeping the number of ads displayed constant as this reduces the potential long-term effects on user behaviour, something not captured at the scale of a rollout. To maximize the expected reward Eθ[R] while satisfying the constraint Eθ[S] = S0, we may add a Lagrangian term to Ĵν(θ) and iteratively solve the following problem:
max θ min α ∑ i R(τi) p(τi|ν) p(τi|θ0) ( 1 + log p(τi|θ) p(τi|ν) ) + α (∑ i S(τi) p(τi|ν) p(τi|θ0) ( 1 + log p(τi|θ) p(τi|ν) ) − S0 ) where α is the Lagrange multiplier associated with the constraint. Due to the approximation, the constraint will not be exactly satisfied for the first few iterations but the convergence of this algorithm guarantees that the constraint will eventually be satisfied.
7.2 EXTENSION TO NON LOG-CONCAVE POLICIES
The results of this paper rely on a log-concavity assumption on the policy which can be too strong a constraint. Indeed, in many cases, the policy depends in a complex manner on the state, for instance through a deep network. However, most of these policies can still be written as a log-concave policy on a non-linear transformation of the states, for instance when the last layer of the deep network uses a softmax. iPoWER can then be used to transform the optimization problem into a simpler, albeit still not concave, maximization problem where the non-concavity of the output of the network has been removed and only remains the non-concavity of the nonlinear transformation of the state.
7.3 CONCLUSION
We proposed a modification to the PoWER algorithm which allows to improve a policy with a reduced number of rollouts. This method is particularly interesting when there are constraints on the number of rollouts, for instance in a robotic environment or when each policy has to be deployed in an industrial production system. We also proposed an extension to existing EM-based methods which allows for the use of control variates, a potentially useful tool to reduce the variance of the estimator. However, several questions remain. In particular, experiments on the Cartpole benchmark indicate that, despite the use of capped importance weights and control variates, as we do more iterations, we might end up in regions of the space with high variance. It is thus important to use additional regularizers, such as normalized weights or penalizing the standard deviation of the estimator. To maintain the simplicity of the overall algorithm, concave lower bounds of these regularizers must also be found, which is still an open problem.
ACKNOWLEDGMENTS
The author would like to thank Vianney Perchet for helpful discussions.

The paper considers the problem of reinforcement learning where the number of policy updates is required to be low. The problem is well motivated and the author provides an interesting modification to the PoWER algorithm, along with variational bounds on the value function (lemmas 3,4) which are interesting in themselves. They also provide numerical results on the cartpole problem and a problem in online advertising with real data. Overall this is a strong, well-written paper. My main reservation is whether it is completely appropriate for ICLR, since the log-concavity assumption the model relies on appear to restrict to simpler models where representations will be not in fact be learned.
Other comments:
- There is a general lack of baselines in the numerical experiment section. I acknowledge this is somewhat of an unusual setting, but even a simple, well-justified baseline would have been welcome. Since cartpole is a relatively simple problem and the advertising dataset is presumably private, perhaps a way to generate a synthetic advertising dataset would have been interesting.
- I was confused by the control variates as constant scalars - are they meant to be constant baselines? And if so, they appear to be treated as hyperparameters -  why are they not learned or estimated?
- There is an interesting section on constrained optimization, but as it is, feels a bit disconnected from the rest of the paper. It appears applicable to the problem of online advertising, but is not mentioned in the corresponding experimental section. Also might be worth adding a citation to the literature of constrained MDPs which develops similar lagrangian ideas.


DIFFERENTIABLE CANONICAL CORRELATION ANALYSIS
Canonical Correlation Analysis (CCA) computes maximally-correlated linear projections of two modalities. We propose Differentiable CCA, a formulation of CCA that can be cast as a layer within a multi-view neural network. Unlike Deep CCA, an earlier extension of CCA to nonlinear projections, our formulation enables gradient flow through the computation of the CCA projection matrices, and free choice of the final optimization target. We show the effectiveness of this approach in cross-modality retrieval experiments on two public image-to-text datasets, surpassing both Deep CCA and a multi-view network with freely-learned projections. We assume that Differentiable CCA could be a useful building block for many multi-modality tasks.
Canonical Correlation Analysis (CCA) computes maximally-correlated linear projections of two modalities. We propose Differentiable CCA, a formulation of CCA that can be cast as a layer within a multi-view neural network. Unlike Deep CCA, an earlier extension of CCA to nonlinear projections, our formulation enables gradient flow through the computation of the CCA projection matrices, and free choice of the final optimization target. We show the effectiveness of this approach in cross-modality retrieval experiments on two public image-to-text datasets, surpassing both Deep CCA and a multi-view network with freely-learned projections. We assume that Differentiable CCA could be a useful building block for many multi-modality tasks.
1 INTRODUCTION
Deep Canonical Correlation Analysis (DCCA) (Andrew et al., 2013) is a non-linear extension of classic Canonical Correlation Analysis (CCA) (Hotelling, 1936) that learns highly correlated latent representations on top of two different neural networks. The central idea of our work is to extend this formulation and cast CCA as a fully differentiable neural network layer which allows for parameter optimization via back-propagation through the CCA projection matrices. This is in contrast to DCCA, where correlation analysis is the topmost part of the network and only used as an optimization target for maximizing the correlation between the respective views. DCCA in general gained a lot of attention recently. It inspired related methods such as Deep Linear Discriminant Analysis (Dorfer et al., 2015) as well as a discriminative re-formulation of DCCA (Elmadany et al., 2016) applied to improve speech-based emotion recognition. Wang et al. (2015a) show that joint optimization of correlation and reconstruction error in auto-encoder configurations is successfully used for representation learning on a multi-modal speech production dataset. We take this as a motivation to evolve and extend the applicability of DCCA.
In our experiments, we employ the proposed differentiable CCA layer in a cross-modality retrieval setup. Cross-modality retrieval is the task of retrieving relevant data of another type when a sample of a different modality is given as a search query. A recent survey by Wang et al. (2016) categorizes the task into binary and real-valued representation learning. In the case of real-valued representation learning, End-to-End DCCA (Yan & Mikolajczyk, 2015) achieves state of the art retrieval results in combination with retrieval by cosine distance computation. With differentiable CCA, it becomes possible to train the networks to directly minimize the objective which will be used for retrieval (e.g., the cosine distance), while still benefitting from the optimally-correlated projections obtained by CCA. Results on two publicly available datasets (Flickr30k (Young et al., 2014), IAPR TC-12
(Grubinger et al., 2006)) suggest that our approach is capable to improve retrieval results in both directions.
The remainder of our paper is structured as follows. In Section 2, we review classic and deep CCA, which are the basis for the differentiable CCA layer proposed in Section 3. In Section 4, we show results of an experimental evaluation in a cross-modality retrieval setting and provide further investigations on the representations learned by our networks. Finally, Section 5 concludes the paper.
2 FOUNDATION: CANONICAL CORRELATION ANALYSIS AND DEEP CCA
In this section, we review the concepts of classical and deep Canonical Correlation Analysis, the basis for the methodology proposed in this work.
2.1 CANONICAL CORRELATION ANALYSIS (CCA)
Let x ∈ Rdx and y ∈ Rdy denote two random vectors with covariances Σxx and Σyy and crosscovariance Σxy . The objective of CCA is to find two matrices A∗ ∈ Rdx×k and B∗ ∈ Rdy×k (with k ≤ dx and k ≤ dy) that project x and y into a common space maximizing their cross-correlation:
(A∗,B∗) = arg max A,B corr(A′x,B′y) (1)
To fix the scale of A and B and force the projected dimensions to be uncorrelated, the optimization is further constrained to A′ΣxxA = B′ΣyyB = I, arriving at:
(A∗,B∗) = arg max A′ΣxxA=B′ΣyyB=I A′ΣxyB (2)
Let T = Σ−1/2xx ΣxyΣ −1/2 yy , and let T = U diag(d)V′ be the Singular Value Decomposition (SVD) of T with ordered singular values di ≥ di+1. As shown by Mardia et al. (1979), we obtain A∗ and B∗ from the top k left- and right-singular vectors of T:
A∗ = Σ−1/2xx U:k B ∗ = Σ−1/2yy V:k (3)
Moreover, the cross-correlation in the projection space is the sum of the top k singular values: corr(A∗′x,B∗′y) = ∑ i≤k di (4)
In practice, the covariances and cross-covariance of x and y are usually not known, but estimated from a training set of m paired vectors, expressed as matrices X ∈ Rdx×m,Y ∈ Rdy×m:
X = X− 1 m X1 Y = Y − 1 m Y1 (5)
Σ̂xx = 1
m− 1 XX′ + rI Σ̂xy =
1
m− 1 XY′ Σ̂yy =
1
m− 1 YY′ + rI (6)
Here, r is a regularization parameter ensuring the matrices are positive definite. Substituting these estimates for Σxx, Σxy and Σyy , respectively, we can estimate A∗ and B∗ using Equation 3.
2.2 DEEP CANONICAL CORRELATION ANALYSIS (DCCA)
Andrew et al. (2013) propose an extension of CCA that allows learning parametric nonlinear transformations of two variables maximizing the cross-correlation after optimal projection. Specifically, let a ∈ Rda and b ∈ Rdb denote two random vectors, and let x = f(a; Θf ) and y = g(b; Θg) denote their nonlinear transformations, parameterized by Θf and Θg . For example, f and g could be feed-forward neural networks. As before, Equation 3 gives the linear transformations of x and y optimizing the CCA objective in Equation 2. Deep CCA optimizes Θf and Θg to further increase the cross-correlation. For dx = dy = k, the CCA objective is equal to the sum of all singular values of T (Equation 4), which is equal to its trace norm:
corr(f(a; Θf ), g(b; Θg)) = corr(x,y) = ||T||tr = tr(T′T)1/2 (7)
Andrew et al. (2013) show how to compute the gradient of this Trace Norm Objective (TNO) with respect to x and y. Assuming f and g are differentiable with respect to Θf and Θg (as is the case for neural networks), this allows to optimize the nonlinear transformations via a gradient-based method. Figure 1a shows a schematic sketch of DCCA, as a fixed objective backpropagated through two neural networks.
3 DIFFERENTIABLE IMPLEMENTATION OF CCA
In this section, we further extend DCCA to allow not only an arbitrary nonlinear transformation of the inputs, but also arbitrary transformations of (or objectives on) the projected vectors. This allows CCA to be used as a building block within a multi-modality neural network, instead of as a final objective only. In the following, we will discuss how to enable backpropagation through CCA, what to consider when doing stochastic updates, and how to apply it for cross-modality retrieval.
3.1 GRADIENT OF CCA
As mentioned above, we can compute the canonical correlation along with the optimal projection matrices from the singular value decomposition T = Σ−1/2xx ΣxyΣ −1/2 yy = Udiag(d)V′. Specifi-
cally, we obtain the correlation as ∑
i di, and projections as A ∗ = Σ −1/2 xx U and B∗ = Σ −1/2 yy V. For
DCCA, it suffices to compute the gradient of the total correlation wrt. x and y in order to backpropagate it through the two networks f and g. Using the chain rule, Andrew et al. (2013) decompose this into the gradients of the total correlation wrt. Σxx, Σxy and Σyy, and the gradients of those wrt. x and y. Their derivations of the former make use of the fact that both the gradient of ∑ i di wrt. T and the gradient of ||T||tr (the trace norm objective in Equation 7) wrt. T′T have a simple form; see Andrew et al. (2013, Sec. 7) for details.
For our differentiable CCA, we instead need the gradients of the projected data A∗′x and B∗′y wrt. x and y, which require ∂U∂x,y and ∂V ∂x,y . We could again decompose this into the gradients wrt. T, the gradients of T wrt. Σxx, Σxy and Σyy and the gradients of those wrt. x and y. However, while the gradients of U and V wrt. T are known (Papadopoulo & Lourakis, 2000), they involve solving O((dxdy)
2) linear 2×2 systems. To arrive at a more practical implementation that does not require the gradient of the SVD, we reformulate the solution to use two symmetric eigendecompositions TT′ = Udiag(e)U′ and T′T = V diag(e)V′ (Petersen & Pedersen, 2012, Eq. 270). This gives us the same left and right eigenvectors we would obtain from the SVD (save for possibly flipped signs, which are easy to fix), along with the squared singular values (ei = d2i ). The gradients of eigenvectors of symmetric real eigensystems have a simple form (Magnus, 1985, Eq. 7) and both
TT′ and T′T are differentiable wrt. x and y, enabling a sufficiently efficient implementation in a graph-based, auto-differentiating math compiler such as Theano (Theano Development Team, 2016).
3.2 STOCHASTIC OPTIMIZATION
For classical CCA, Σxx, Σxy and Σyy are estimated from a large set of m training examples (Equation 6). In contrast, gradient-based optimization of neural networks usually estimates the gradients wrt. network parameters from mini-batches of n randomly drawn examples, with n m. In Deep CCA as well as in our extension, the correlations are functions of the network parameters that we need to backpropagate through, effectively enforcing m = n.
Andrew et al. (2013) solve this discrepancy by optimizing the network parameters with L-BFGS on the full training set, which is infeasible for very large datasets. Yan & Mikolajczyk (2015) instead train on small mini-batches, estimating correlation matrices of size 4096×4096 from 100 examples only, which seems risky. We will choose a way in between, training on large mini-batches to obtain stable estimates. This approach was also taken by Wang et al. (2015b, Sec. 5.1), who found minibatches of 400–1000 examples to even outperform full-batch L-BFGS. In addition, for testing, we optionally re-estimate the correlation matrices (and the corresponding projection matrices) using a larger set of m > n examples.
Another tempting option is to train on small mini-batches, but use exponential moving averages updated with each mini-batch as follows:
Σxx ← Σxx(1− α) + Σ̂xxα Σxy ← Σxy(1− α) + Σ̂xyα Σyy ← Σyy(1− α) + Σ̂yyα With proper initialization and a sufficiently small coefficient α, this gives stable estimates even for small n. However, since only the estimates from the current mini-batch Σ̂xx, Σ̂xy and Σ̂yy can be practically considered in backpropagation, this changes the learning dynamics: For too small α, the projection matrices will be virtually degraded to constants. Empirically, we found that large mini-batches perform slightly better than small batches with moving averages (see Appendix B).
3.3 CROSS-MODALITY RETRIEVAL WITH DIFFERENTIABLE CCA
DCCA maximizes the correlation between the latent representations of two different neural networks. When the two network inputs a and b represent different views of an entity (e.g., an image and its textual description), DCCA projects them into a common space where they are highly correlated. This can be exploited for cross-modality retrieval: Projecting one modality of an entity, we can find the best-matching representations of the second modality (e.g., an image for a textual description, or vice versa). To find the best matches, a common option is to compute nearest neighbors in terms of cosine distance (Yan & Mikolajczyk, 2015), which is closely related to correlation.
Given the methodology introduced above, we now have the means to optimize DCCA projections directly for the task at hand. In Figure 1b, we show a possible setting where we put the differentiable CCA layer on top of a multi-view network. Instead of optimizing the networks to maximize the correlation of the projected views (the TNO), we can optimize the networks towards a task-specific objective and still benefit from the optimality of the CCA projections.
For this work, we optimize towards minimal cosine distance between the correlated views, the very metric used for retrieval. In the next section, we empirically show that this is indeed beneficial in terms of quantitative retrieval performance as well as convergence speed of network training.
4 EXPERIMENTS
We evaluate our approach in cross-modality retrieval experiments on two publicly available datasets (also considered by Yan & Mikolajczyk (2015)) and provide investigations on the representations learned by the network.
4.1 EXPERIMENTAL SETUP
For the evaluation of our approach, we consider Flickr30k and IAPR TC-12, two publicly available datasets for cross-modality retrieval. Flickr30k consists of image-caption pairs, where each image
is annotated with five different textual descriptions. The train-validation-test split for Flickr30k is 28000-1000-1000. In terms of evaluation setup, we follow the related work and report results on two different evaluation protocols. Protocol pooled pools the five available captions into one "concatenated" text, meaning that only one but richer text annotation remains per image. This is done for all three sets. Protocol 5 captions pools only the captions of the train set and keeps five separate annotations for validation and test set. The IAPR TC-12 dataset contains 20000 natural images where only one – but compared to Flickr30k more detailed – caption is available for each image. As no predefined train-validation-test split is provided, we randomly select 2000 images for testing, 1000 for validation and keep the rest for training. Yan & Mikolajczyk (2015) also use 2000 images for testing, but did not explicitly mention hold out images for validation. Table 1 shows an example image along with its corresponding captions or caption for either dataset.
The task at hand for both datasets is to retrieve the correct counterpart – either text or image – when given a query element of the other modality. We follow Yan & Mikolajczyk (2015) and use the cosine distance for retrieval in the projection space. As evaluation measures we consider the Recall@k (R@k) as well as the Median Rank (MR) and the Mean Average Precision (MAP). The R@k rate (high is better) is the ratio of queries which have the correct corresponding counterpart in the first k retrieval results. The MR is the median position (low is better) of the target in a similarityordered list of available candidates. Finally, we define the MAP (high is better) as the mean value of 1/Rank over all queries.
The input to our networks is a 4096-dimensional image feature vector along with a corresponding text vector representation (5793 for Flickr30k, 2048 for IAPR TC-12). In terms of text preprocessing, we follow Yan & Mikolajczyk (2015), tokenizing and lemmatizing the raw captions as the first step. Based on the lemmatized captions, we compute l2-normalized TF/IDF-vectors, omitting words with an overall occurrence smaller than 5 times for Flickr30k and 3 times for IAPR TC-12, respectively. The image represenations are computed from the last hidden layer of a network pretrained on ImageNet (layer fc7 of CNN_S by Chatfield et al. (2014)).
4.2 NETWORK ARCHITECTURES AND OPTIMIZATION DETAILS
We feed 4096-dimensional image vectors along with the corresponding text representation into our networks. The image representation is followed by a linear dense layer with 128 units (this will also be the dimensionality k = 128 of the resulting CCA retrieval space). The text vector is processed by two batch-normalized (Ioffe & Szegedy, 2015) dense layers of 1024 units each and an ELU activation function (Clevert et al., 2015). As a last layer for the text representation network, we again apply a dense layer with 128 linear units. For a fair comparison, we keep the structure (and number of parameters) of all networks in our experiments the same. The only parameters that vary are the objectives and the corresponding optimization/regularization strategies. In particular, we apply a grid search on the respective hyper-parameters and report the best results for each method. Optimization is performed either using Stochastic Gradient Descent (SGD) with momentum or by the adam (Kingma & Ba, 2014) update rule.
As optimization targets, we consider the following candidates: (1) The Trace Norm Objective (TNO) as our base line for cross-modality retrieval (Yan & Mikolajczyk, 2015). (2) The proposed differentiable CCA layer in combination with the objectives cosine distance (CCAL-cos), squared cosine distance (CCAL-cos2) and euclidean distance (CCAL-l2). As an additional setting, we consider a freely-learnable projection layer where the projection matrices A and B are randomly initialized weights that can be optimized by the network using SGD in the conventional way. This allows to assess the benefit of using CCA-derived projections within a multi-view network under otherwise unchanged objectives. For this experiment, we optimize for the squared cosine distance and denote the setting by learned-cos2. The batch size is set to 1000 samples to allow stable covariance estimates for the CCA (Section 3.2). For further stabilization, we regularize the covariance matrices (Andrew et al., 2013) by adding scaled (r = 10−3) identity matrices to the estimates Σxx, Σyy and T (Section 2.1). The variants based on differentiable CCA are additionally regularized by L2 weight decay. No dropout is used in this settings as it harmed optimization in our experiments. When optimizing with the TNO we follow Yan & Mikolajczyk (2015) and use dropout (p = 0.5) after the first two dense layers of the text network. In Table 4 in Appendix A we provide the optimization settings for all configurations in detail, found using a grid search optimizing MAP on the validation set.
4.3 EXPERIMENTAL RESULTS ON CROSS-MODALITY RETRIEVAL
Table 2 lists our results on Flickr30k. Along with our experiments, we also show the results reported in (Yan & Mikolajczyk, 2015) as a reference (E2E-DCCA). However, a direct comparison to our results may not be fair: E2E-DCCA uses a different ImageNet-pretrained network for the image representation, and finetunes this network while we keep it fixed (as we are only interested in comparing differentiable CCA to alternatives, not in obtaining the best possible results). Our TNO results use the same objective as E2E-DCCA, but our network architecture, permitting direct comparison.
When comparing the performance of our networks, we observe a gain both for image-to-text and text-to-image retrieval when training with the CCAL-cos2 objective compared to TNO (e.g., R@1 of 34.1 compared to 29.9 under protocol pooled). This indicates that training a network directly on the objective used for retrieval (using differentiable CCA) is a reasonable design choice. A closer look at the results also reveals that the squared cosine distance is superior compared to the remaining objectives. We further observe that the randomly initialized projection matrices learned entirely by SGD (learned-cos2) show poor performance compared to their CCA counterpart (even though in theory, they could converge to exactly the same solution). This suggests that exploiting the beneficial properties of the CCA projections directly within a network during training is a powerful tool, supporting optimization of related objectives. CCAL-l2 for example performs poorer than the variants including cosine losses but still better than the version with learned weights. On protocol
5 captions, we only report the best results (CCAL-cos2) along with the TNO and observe similar tendencies. Note that there are various other methods reporting results on Flickr30k (Karpathy et al., 2014; Socher et al., 2014; Mao et al., 2014; Kiros et al., 2014) which partly surpass ours, for example by using more elaborate processing of the textual descriptions. We omit these results as we focus on the comparison of DCCA with the proposed differentiable CCA layer.
In Table 3, we list our results on the IAPR TC-12 dataset. We again show the retrieval performances of Yan & Mikolajczyk (2015) as a baseline (again with limited comparability, due to a different architecture and a different train-validation-test split), along with our implementation of the TNO and the CCA layer trained with squared cosine distance. For image-to-text retrieval, we achieve slightly better retrieval performances when training with cosine distance and propagating the gradients back through the differentiable CCA layer. For the other direction, results are slightly worse.
4.4 INVESTIGATIONS ON LEARNED REPRESENTATIONS
In this section, we provide a more detailed look at the learned representations. We compare the representations learned with the TNO to the proposed CCA layer optimized with the squared cosine distance objective. For easier comparison, we re-train both networks with a reduced projection dimensionality of h = 64 – otherwise, the TNO takes much longer to converge than the CCA layer. This results in slightly decreased performance for both, but the relative tendences are preserved.
Figure 2a shows the evolution of the mean correlation (mean over singular values with maximum 1.0) on the training set during optimization. Allong with the correlation, we also plot the average cosine distance between corresponding pairs on the validation set. As expected, for the TNO we observe a continous decrease of cosine distance when the correlation increases. Interestingly, this is not the case for CCAL-cos2. The result suggests that the network found a way of minimizing the cosine distance other than by increasing correlation between the representations – the latter even decreases after a few training epochs. In Figure 2b, we plot the corresponding evolution of MAP on the training and validation set, confirming that the decreased cosine distance indeed also leads to improved retrieval performance. Finally, in Figure 2c we compare the individual correlation coefficients (magnitudes of CCA singular values on the training set) of both representations after the last training epoch. This details the observation in Figure 2a: not only the total correlation, but also the individual correlation coefficients are considerably higher when training with TNO, even though the retrieval performance is lower.
5 CONCLUSION
We presented a fully differentiable version of Canonical Correlation Analysis which enables us to back-propagate errors directly through the computation of CCA. As this requires to establish gradient flow through CCA, we formulate it to allow easy computation of the partial derivatives ∂A∗
∂x,y and ∂B∗ ∂x,y of CCA’s projection matrices A ∗ and B∗ with respect to the input data x and y. With this formulation, we can incorporate CCA as a building block within multi-modality neural networks that produces maximally-correlated projections of its inputs. In our experiments, we use this building block within a cross-modality retrieval setting, optimizing a network to minimize the cosine distance of the correlated CCA projections. Experimental results show that when using the cosine distance for retrieval (as is common for correlated views), this is superior to optimizing a network for maximally-correlated projections (as done in Deep CCA), or not using CCA at all. We further observed (Section 4.4) that it is not necessarily required to have maximum correlation to achieve a high retrieval performance. Finally, our differentiable CCA layer could provide a useful basis for further research, e.g., as an intermediate processing step for learning binary cross-modality retrieval representations.
ACKNOWLEDGMENTS
The research reported in this paper has been supported by the Austrian Federal Ministry for Transport, Innovation and Technology, the Federal Ministry of Science, Research and Economy, and the Province of Upper Austria in the frame of the COMET center SCCH, as well as by the Federal Ministry for Transport, Innovation & Technology (BMVIT) and the Austrian Science Fund (FWF): TRP 307-N23. The Tesla K40 used for this research was donated by the NVIDIA Corporation.
APPENDIX A: OPTIMIZATION SETTINGS
The table below provides a detailed listing of the optimization strategies for all our experiments. All our configurations are of course also available in our experimental code published at (will be added).
APPENDIX B: INFLUENCE OF RUNNING AVERAGE STATISTICS
In this additional section, we investigate the influence of weighting coefficient α when using exponential moving average estimates of the covariance matrices for CCA computation (see Section 3). A high α (close to 1.0) means that the averaged estimate of Σxx, Σyy and Σxy mostly depends on the current batch, and a low α (close to 0.0) means it more strongly depends on the history of previous batches. To assess whether and under what circumstances exponential moving averages are helpful, we run an additional experiment on the IAPR TC-12 dataset as follows: We re-train one of the models of Section 4 both with batch size 1000 and with batch size 200, varying α from 1.0 to 0.1 with a step size of 0.1 and measuring the MAP achieved on the validation set. We run each setting three times and report the average over the three runs. Figure 3 shows the results of this experiment. For batch size 1000, we draw the same conclusion as was reported in (Wang et al., 2015a;b): If the batch size is sufficiently large and representative for the entire population, learning on distribution parameters (in this case covariance matrices) is feasible, and the network performs best when trained with an α close to one. This is not the case for batch size 200. In particular, the configurations with a large α (small effective running average window) perform poorly. We conclude that a batch size of 200 is too small to obtain stable and representative covariances. However, when choosing a small α, it is still possible to train the models and achieve reasonable retrieval performance. As a practical recommendation, we suggest to use large batch sizes whenever possible (e.g., if feasible with available hardware). If the batch size needs to be reduced (e.g., for very large models and limited memory), using small alpha values still allows for training canonically correlated retrieval networks. For this work, we use a batch size of 1000 and fix α = 1, disabling moving averages.

The authors propose to combine a CCA objective with a downstream loss.  This is a really nice and natural idea.  However, both the execution and presentation leave a lot to be desired in the current version of the paper.
It is not clear what the overall objective is.  This was asked in a pre-review question but the answer did not fully clarify it for me.  Is it the sum of the CCA objective and the final (top-layer) objective, including the CCA constraints?  Is there some interpolation of the two objectives?
By saying that the top-layer objective is "cosine distance" or "squared cosine distance", do you really mean you are just minimizing this distance between the matched pairs in the two views?  If so, then of course that does not work out of the box without the intervening CCA layer:  You could minimize it by setting all of the projections to a single point.  A better comparison would be against a contrastive loss like the Hermann & Blunsom one mentioned in the reviewer question, which aims to both minimize the distance for matched pairs and separate mismatched ones (where "mismatched" ones can be uniformly drawn, or picked in some cleverer way).  But other discriminative top-layer objectives that are tailored to a downstream task could make sense.
There is some loose terminology in the paper.  The authors refer to the "correlation" and "cross-correlation" between two vectors.  "Correlation" normally applies to scalars, so you need to define what you mean here.  "Cross-correlation" typically refers to time series.  In eq. (2) you are taking the max of a matrix.  Finally I am not too sure in what way this approach is "fully differentiable" while regular CCA is not -- perhaps it is worth revisiting this term as well.
Also just a small note about the relationship between cosine distance and correlation:  they are related when we view the dimensions of each of the two vectors as samples of a single random variable.  In that case the cosine distance of the (mean-normalized) vectors is the same as the correlation between the two corresponding random variables.  In CCA we are viewing each dimension of the vectors as its own random variable.  So I fear the claim about cosine distance and correlation is a bit of a red herring here.
A couple of typos:
"prosed" --> "proposed"
"allong" --> "along"


PARAMETRIC EXPONENTIAL LINEAR UNIT FOR DEEP CONVOLUTIONAL NEURAL NETWORKS
The activation function is an important component in Convolutional Neural Networks (CNNs). For instance, recent breakthroughs in Deep Learning can be attributed to the Rectified Linear Unit (ReLU). Another recently proposed activation function, the Exponential Linear Unit (ELU), has the supplementary property of reducing bias shift without explicitly centering the values at zero. In this paper, we show that learning a parameterization of ELU improves its performance. We analyzed our proposed Parametric ELU (PELU) in the context of vanishing gradients and provide a gradient-based optimization framework. We conducted several experiments on CIFAR-10/100 and ImageNet with different network architectures, such as NiN, Overfeat, All-CNN and ResNet. Our results show that our PELU has relative error improvements over ELU of 4.45% and 5.68% on CIFAR-10 and 100, and as much as 7.28% with only 0.0003% parameter increase on ImageNet. We also observed that Vgg using PELU tended to prefer activations saturating closer to zero, as in ReLU, except at the last layer, which saturated near -2. Finally, other presented results suggest that varying the shape of the activations during training along with the other parameters helps controlling vanishing gradients and bias shift, thus facilitating learning.
1 INTRODUCTION
Over the past few years, Convolutional Neural Networks (CNNs) have become the leading approach in computer vision (Krizhevsky et al., 2012; LeCun et al., 2015; Vinyals et al., 2015; Jaderberg et al., 2015; Ren et al., 2015; Hosang et al., 2016). Through a series of non-linear transformations, CNNs can process high-dimensional input observations into simple low-dimensional concepts. The key principle of CNNs is that features at each layer are composed of features from the layer below. This creates a hierarchical organization of increasingly abstract concepts. Since levels of organization are often seen in complex biological structures, such a hierarchical organization makes CNNs particularly well-adapted for capturing high-level abstractions from real-world observations.
The activation function plays a crucial role in learning representative features. Defined as max{h, 0}, the Rectified Linear Unit (ReLU) is one of the most popular activation function (Nair & Hinton, 2010). It has interesting properties, such as low computational complexity, non-contracting first-order derivative and induces sparse activations, which have been shown to improve performance Krizhevsky et al. (2012). The main drawback of ReLU is its zero derivative for negative arguments. This blocks the back-propagated error signal from the layer above, which may prevent the network from reactivating dead neurons. To overcome this limitation, Leaky ReLU (LReLU) adds a positive slope a to the negative part of ReLU (Maas et al., 2013). Defined as max{h, 0}+ amin{h, 0}, where a > 0, LReLU has a non-zero derivative for negative arguments. Unlike ReLU, its parameter a allows a small portion of the back-propagated error signal to pass to the layer below. By using a small enough value a, the network can still output sparse activations while preserving its ability to reactivate dead neurons. In order to avoid specifying by hand the slope parameter a, Parametric ReLU (PReLU) directly learns its value during back-propagation (He et al., 2015b). As the training phase progresses, the network can adjust its weights and biases in conjunction with the slopes a of all its PReLU for
potentially learning better features. Indeed, He et al. (2015b) have empirically shown that learning the slope parameter a gives better performance than manually setting it to a pre-defined value.
A recently proposed important activation function is the Exponential Linear Unit (ELU). Is is defined as identity for positive arguments and a(exp(h) − 1) for negative ones (Clevert et al., 2015). The parameter a can be any positive value, but is usually set to 1. ELU has the interesting property of reducing bias shift, which is defined as the change of a neuron’s mean value due to weight update. If not taken into account, bias shift leads to oscillations and impeded learning (Clevert et al., 2015). Clevert et al. (2015) have shown that either centering the neuron values at zero or using activation functions with negative values can reduce bias shift. Centering the neuron values can be done with the Batch Normalization (BN) method (Ioffe & Szegedy, 2015), while adding negative values can be done with parameterizations such as LReLU or PReLU.
Based on the observation that learning a parameterization of ReLU improves performance (He et al., 2015b), we propose the Parametric ELU (PELU) that learns a parameterization of ELU. We define parameters controlling different aspects of the function and propose learning them during back-propagation. Our parameterization preserves differentiability by acting on both the positive and negative parts of the function. Differentiable activation functions usually give better parameter updates during back-propagation (LeCun et al., 2015). PELU also has the same computational complexity as ELU. Since parameters are defined layer-wise instead of per-neurons, the number of added parameters is only 2L, where L is the number of layers. Our experiments on the CIFAR-10/100 and ImageNet datasets have shown that ResNet (Shah et al., 2016), Network in Network (Lin et al., 2013), All-CNN (Springenberg et al., 2015) and Overfeat (Sermanet et al., 2013) with PELU all had better performances than with ELU. We finally show that our PELUs in the CNNs adopt different non-linear behaviors during training, which we believe helps the CNNs learning better features.
The rest of the paper is organized as follows. We present related works in section 2 and described our proposed approach in section 3. We detail our experimentations in section 4 and discuss the results in section 5. We conclude the paper in section 6.
2 RELATED WORK
Our proposed PELU activation function is related to other parametric approaches in the literature. The Adaptive Piecewise Linear (APL) unit learns a weighted sum of S parametrized Hinge functions (Agostinelli et al., 2014). One drawback of APL is that the number of points at which the function is non-differentiable increase linearly with S. Moreover, though APL can be either a convex or non-convex function, the rightmost linear function is forced to have unit slope and zero bias. This may be an inappropriate constraint which could affect the representation ability of the CNNs.
Another activation function is Maxout, which outputs the maximum over K affine functions for each input neuron (Goodfellow et al., 2013). The main drawback of Maxout is that it multiplies by K the amount of weights to be learned in each layer. For instance, in the context of CNNs, we would apply a max operator over the feature maps of each K convolutional layers. This could become too computationally demanding in cases where the CNNs are very deep. Unlike Maxout, our PELU adds only 2L parameters, where L is the number of layers.
Finally, the S-Shaped ReLU (SReLU) imitates the Webner-Fechner law and the Stevens law by learning a combination of three linear functions (Jin et al., 2015). Although this parametric function can be either convex or non-convex, SReLU has two points at which it is non-differentiable. Unlike SReLU, our PELU is fully differentiable, since our parameterization acts on both the positive and negative sides of the function. This in turn improves the back-propagation weight and bias updates.
3 PARAMETRIC EXPONENTIAL LINEAR UNIT (PELU)
In this section, we present our proposed PELU function and analyze it in the context of vanishing gradients. We also elaborate on the gradient descent rules for learning the parameterization.
3.1 DEFINITION
The standard Exponential Linear Unit (ELU) is defined as identity for positive arguments and a(exp(h)− 1) for negative arguments (Clevert et al., 2015). Although the parameter a can be any positive value, Clevert et al. (2015) proposed using a = 1 to have a fully differentiable function. For other values a 6= 1, the function is non-differentiable at h = 0. For this reason, we do not directly learn parameter a during back-propagation. Updating a with the gradient would break differentiability at h = 0, which could imped back-propagation.
We start by adding two additional parameters to ELU as follows:
f(h) = { ch if h ≥ 0 a(exp(hb )− 1) if h < 0 , a, b, c > 0 , (1)
for which the original ELU can be recovered when a = b = c = 1. As shown in Figure 1, each parameter in (1) controls different aspects of the activation. Parameter c changes the slope of the linear function in the positive quadrant (the larger c, the steeper the slope), parameter b affects the scale of the exponential decay (the larger b, the smaller the decay), while a acts on the saturation point in the negative quadrant (the larger a, the lower the saturation point). We also constrain the parameters to be positive to have a monotonic function. Consequently, reducing the weight magnitude during training always lowers the neuron contribution.
Using this parameterization, the network can control its non-linear behavior throughout the course of the training phase. It may increase the slope with c or the decay with b to counter vanishing gradients, and push the mean activation towards zero by lowering the saturation point with a for better managing bias shift. We now look into gradient descent and define update rules for each parameter, so that the network can adjust its behavior as it seems fit. However, a standard gradient update on parameters a, b, c would make the function non-differentiable at h = 0 and impair back-propagation. Instead of relying on a projection operator to restore differentiability after each update, we constrain our parameterization by forcing f to stay differentiable at h = 0. We equal the derivatives on both sides of zero, and solve for c:
∂ch
∂h ∣∣∣∣ h=0 = ∂a(exp(hb )− 1) ∂h ∣∣∣∣∣ h=0
(2)
which gives c = ab as solution. Incorporating (2) gives the proposed Parametric ELU (PELU): PELU: f(h) = {
a bh if h ≥ 0 a(exp(hb )− 1) if h < 0 , a, b > 0 (3)
With this parameterization, in addition to changing the saturation point and exponential decay respectively, both a and b adjust the slope of the linear function in the positive part to ensure differentiability at h = 0.
3.2 ANALYSIS
To understand the effect of the proposed parameterization, we now investigate the vanishing gradient for the following simple network, containing one neuron in each of its L layers:
x = h0, hl = wlhl−1, zl = f(hl), E = `(zL, y) (1 ≤ l ≤ L) (4)
where we have omitted, without loss of generality, the biases for simplicity. In (4), ` is the loss function between the network prediction zL and label y, which takes value E at x. In this case, it can be shown using the chain rule of derivation that the derivative of E with respect to any weight k is:
∂E
∂wk = hk−1f
′(hk)  L∏ j=k+1 f ′(hj)wj  ∂E ∂zL ,
where f ′(hk) is a shortcut for ∂zk∂hk . Vanishing gradient happens when the product term inside the bracket has a very small magnitude, which makes ∂E∂wk ≈ 0. Since the updates are proportional to the gradients, the weights at lower layers converge more slowly than those at higher layers, due to the exponential decrease as k gets smaller (the product has L− k terms). One way the network can fight vanishing gradients is with f ′(hj)wj ≡ f ′(wjhj−1)wj ≥ 1, so that the magnitude of the product does not tend to zero. Therefore, a natural way to investigate vanishing gradient is by analyzing the interaction between weight w and activation h, after dropping layer index j. Specifically, our goal is to find the range of values h for which f ′(wh)w ≥ 1. This will indicate how precise the activations h must be to manage vanishing gradients. Theorem 1. If w ≥ ba and h < 0, then w
∗ = exp(1) ba maximizes the interval length of h for which f ′(wh)w ≥ 1, which length takes value l∗ = a exp(−1).
Proof. With our proposed PELU, we have:
f ′(wh)w = { w ab if h ≥ 0 w ab exp(wh/b) if h < 0 , a, b > 0 (5)
Assuming w ≥ ba and h < 0, and using the fact that w a b exp(wh/b) is monotonically increasing, the interval length l(w) of values h for which w ab exp(wh/b) ≥ 1 is given by the magnitude of the zero of w ab exp(wh/b) − 1. Solving the derivative equals zero for h gives l(w) = | log( b a 1 w )|( b w ). Using the fact that w ≥ ba , it can be shown that l(w) is pseudo-concave, so it has a unique optimum. Maximizing l(w) with respect to w is thus the solution of solving the derivative equals zero, which gives l∗ = a exp(−1), at w∗ = exp(1) ba .
This result shows that in the optimal scenario where w = exp(1) ba , the length of negative values h for which f ′(wh)w ≥ 1 is no more than a exp(−1). Without our proposed parameterization (a, b = 1), dealing with vanishing gradient is mostly possible with positive arguments, which makes the negative ones (useful for bias shift) hurtful for back-propagation. With the proposed parameterization, a can be adjusted to increase the length a exp(−1) and allow more negative activations h to counter vanishing gradients. The ratio ba can also be modified to ensure w ≥ b a so that f
′(wh)w ≥ 1 for h > 0. Based on this analysis, the proposed parameterization gives more flexibility to the network, and the experiments in Section 4 have shown that the networks do indeed take advantage of it.
3.3 OPTIMIZATION
PELU is trained simultaneously with all the network parameters during back-propagation. Using the chain rule of derivation, the derivative of objective E with respect to a and b for one layer is:
∂E ∂a = ∑ i ∂E ∂f(hi) ∂f(hi) ∂a ,
∂E ∂b = ∑ i ∂E ∂f(hi) ∂f(hi) ∂b , (6)
where i sums over all elements of the tensor on which f is applied. The terms ∂E∂f(hi) are the gradients propagated from the above layers, while ∂f(h)∂a and ∂f(h) ∂b are the gradients of f with respect to a, b:
∂f(h)
∂a = { h b if h ≥ 0 exp(h/b)− 1 if h < 0 ,
∂f(h)
∂b = { −ahb2 if h ≥ 0 − ab2 exp(h/b) if h < 0 . (7)
To preserve the parameter positivity after the updates, we force them to always be greater than 0.1. The update rules are the following:
∆a← µ∆a− α∂E ∂a
a← max {a+ ∆a, 0.1} ,
∆b← µ∆b− α∂E ∂b
b← max {b+ ∆b, 0.1} (8)
In (8), µ is the momentum and α is the learning rate. When specified by the training regimes, we also use a `2 weight decay regularization on both the weight matrices W and PELU parameters. This is different than PReLU, which did not use weight decay to avoid a shape bias towards ReLU. In our case, weight decay is necessary for a and b, otherwise the network could circumvent it for the W s by adjusting a or b, a behavior that would be hurtful for training.
4 EXPERIMENTATIONS
In this section, we present our experiments in supervised learning on the CIFAR-10/100 and ImageNet tasks. Our goal is to show that, with the same network architecture, parameterizing ELU improves the performance. We also provide results with the ReLU activation function for reference.
4.1 CIFAR-10/100
As first experiment, we performed object classification on the CIFAR-10 and CIFAR-100 datasets (60,000 32x32 colored images, 10 and 100 classes respectively) (Krizhevsky et al., 2012). We trained a residual network (ResNet) with the identity function for the skip connexion and bottleneck residual mappings with shape (Conv + ACT)x2 + Conv + BN (He et al., 2015a; Shah et al., 2016). The ACT module is either PELU, ELU or BN+ReLU. We follow Facebook’s Torch implementation fb.resnet.torch1 for data augmentation and learning rate schedule, so that training is not biased towards PELU to the detriment of the other activations.
We also evaluated our proposed PELU on a smaller convolutional network. We refer to this network as SmallNet. It contains three convolutional layers followed by two fully connected layers. The convolutional layers were respectively composed of 32, 64 and 128 3x3 filters with 1x1 stride and 1x1 zero padding, each followed by ACT, 2x2 max pooling with a stride of 2x2 and dropout with probability 0.2. The fully connected layers were defined as 2048→ 512, followed by ACT, dropout with probability 0.5, and a final linear layer 512→ 10 for CIFAR-10 and 512→ 100 for CIFAR-100. We performed global pixel-wise mean subtraction, and used horizontal flip as data augmentation.
Table 1 presents the test error results (in %) of SmallNet and ResNet110 on both tasks, with ELU, ReLU and PELU. For SmallNet, PELU reduced the error of ELU from 14.81% to 13.54% on CIFAR10, and from 39.76% to 38.93% on CIFAR-100, which corresponds to a relative improvement of 8.58% and 2.09% respectively. As for ResNet110, PELU reduced the error of ELU from 5.62% to 5.37% on CIFAR-10, and from 26.55% to 25.04% on CIFAR-100, which corresponds to a relative improvement of 4.45% and 5.68% respectively. These results suggest that parameterizing the ELU activation improves its performance.
It is worth noting for ResNet110 that weight decay played an important role in obtaining these performances. Preliminary experiments conducted with a weight decay of 0.0001 showed no significant improvements of PELU over ELU. We observed larger differences between the train and test set error percentages, which indicated possible over-fitting. By increasing the weight decay to 0.001, we obtained the performance improvements shown in Table 1. Importantly, we did not have to increase the weight decay for SmallNet. The PELU, ELU and BN+ReLU SmallNets used the same decay. Although these results suggest that residual networks with PELU activations may be more prone to over-fitting, weight decay can still be used to correctly regularize the ResNets.
1 https://github.com/facebook/fb.resnet.torch
4.2 IMAGENET
We finally tested the proposed PELU on ImageNet 2012 task (ILSVRC2012) using four different network architectures: ResNet18 (Shah et al., 2016), Network in Network (NiN) (Lin et al., 2013), All-CNN (Springenberg et al., 2015) and Overfeat (Sermanet et al., 2013). We used either PELU, ELU or BN+ReLU for the activation module. Due to NiN’s relatively complex architecture, we added BN after each max pooling layer (every three layers) for further reducing vanishing gradients. Each network was trained following Chintala’s Torch implementation imagenet-multiGPU.torch 2 with the training regimes shown in Table 2. Regime #1 starts at a higher learning rate (1e-1) than regime #2 (1e-2), and has a larger learning rate decay of 10 compared to 2 and 5.
Figure 2 presents the TOP-1 error rate (in %) of all four networks on ImageNet 2012 validation dataset. We see from these figures that PELU consistently obtained the lowest error rates for all networks. The best result was obtained with NiN. In this case, PELU improved the error rate from 40.40% (ELU) to 36.06%, which corresponds to a relative improvement of 7.29%. Importantly, NiN obtained these improvements at little computational cost. It only added 24 additional parameters, i.e. 0.0003% increase in the number of parameters. This suggests that PELU acts on the network in a different manner than the weights and biases. Such a low number of parameters cannot significantly increase the expressive power of the network. We would not have seen such a large improvement by adding 24 additional weights to a convolutional layer with the ELU activation.
2 https://github.com/soumith/imagenet-multiGPU.torch
We see from the curves in Figure 2 that the training regime has an interesting effect on the convergence of the networks. The performance of PELU is closer to the performance of ELU for regime #2, while it is significantly better than ELU for regime #2. Although we do not have a clear explanation to why this is the case, we believe that the small initial learning rate of regime #2 affects PELU optimization due to smaller gradient steps. We also see that the error rates of All-CNN and Overfeat with PELU increase by a small amount starting at epoch 44. Since ELU and ReLU do not have this error rate increase, this shows possible over-fitting for PELU. For regime #2, the error rates decrease more steadily and monotonically. Although performing more experiments would improve our understanding, these results suggest that PELU could and should be trained with larger learning rates and decays for obtaining better performance improvements.
5 DISCUSSION
In this section, we elaborate on other parameter configurations and perform a visual evaluation of the parameter progression throughout the training phase.
5.1 PARAMETER CONFIGURATION
The proposed PELU activation function (3) has two parameters a and b, where a is used with a multiplication and b with a division. A priori, any of the four configurations (a, b), (a, 1b ), ( 1 a , b) or ( 1a , 1 b ) could be used as parameterization. In this section, we show experimentally that PELU with the proposed (a, 1b ) configuration is the preferred choice, as it achieves the best overall results.
For evaluating the effect of parameter configuration, we trained several CNNs on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2012). We used ResNets with a depth varying from 20, 32, 44, 56 to 110. The ResNets had the identity function for the skip connexion and two different residual mappings (He et al., 2015a; Shah et al., 2016). We used a basic block Conv + PELU + Conv + BN block for depth 20, 32 and 44, and bottleneck block (Conv + PELU)x2 + Conv + BN for depth 56 and 110. We report the averaged error rate achieved over five tries.
First, we can see from the results presented in Figure 3 that the error rate reduces as the network gets deeper. This is in conformity with our intuition that deeper networks have more representative capability. Also, we can see that the proposed configuration (a, 1b ) obtained the best performance overall. Configuration (a, 1b ) obtained 5.37% error rate on CIFAR-10 and 25.04% error rate on CIFAR-100. We believe this improvement is due to weight decay. When using configuration (a, 1b ) along with weight decay, pushing parameters a and b towards zero encourages PELU to be similar to ReLU. In this case, the CNN is less penalized for using ReLU and more penalized for using other parametric forms. This may help the CNN to use as much PELUs that look like ReLUs as it needs without incurring a large penalty. The experiments in section 5.2 partly supports our claim. Although the performance improvement of (a, 1b ) is relatively small in comparison to the other three configurations, configuration (a, 1b ) should be preferred for subsequent experiments.
5.2 PARAMETER PROGRESSION
We perform a visual evaluation of the non-linear behaviors adopted by a Vgg network during training (Simonyan & Zisserman, 2014). To this effect, we trained a Vgg network with PELU activations on the CIFAR-10 dataset. We performed global pixel-wise mean subtraction, and used horizontal flip as data augmentation. The trained network obtained 6.95% and 29.29% error rate on CIFAR-10 and CIFAR-100 respectively.
Figure 3 shows the progression of the slope (ab ) and the negative of the saturation point (parameter a) for PELU at layers 2, 7, 10 and 14. We can see different behaviors. In layer 2, the slope quickly increases to a large value (around 6) and slowly decreases to its convergence value. We observe a similar behavior for layer 7, except that the slope increases at a later iteration. Layer 2 increases at about iteration 650 while layer 7 at about iteration 1300. Moreover, in contrast to layer 2 and 7, the slope in layer 10 increases to a smaller value (around 2) and does not decrease after reaching it. Layer 14 also displays a similar behavior, but reaches a much higher value (around 15). We believe that adopting these behaviors helps early during training to disentangle redundant neurons. Since peak activations scatter the inputs more than flat ones, spreading neurons at the lower layers may allow the network to unclutter neurons activating similarly. This may help the higher layers to more easily find relevant features in the data.
The saturation point in layer 2, 7 and 10 converges in the same way to a value near zero, while in layer 14 it reaches a value near -2. This is an interesting behavior as using a negative saturation reduces bias shift. In another experiment with Vgg, we tried adding BN at different locations in the network. We saw similar convergence behaviors for the saturation point. It seems that, it this specific case, the network could counter bias shift with only the last layer, and favored sparse activations in the other layers. These results suggest that the network takes advantage of the parameterization by using different non-linear behaviors at different layers.
6 CONCLUSION
The activation function is a key element in Convolutional Neural Networks (CNNs). In this paper, we proposed learning a parameterization of the Exponential Linear Unit (ELU) function. Our analysis of our proposed Parametric ELU (PELU) suggests that CNNs with PELU may have more control over bias shift and vanishing gradients. We performed several supervised learning experiments and showed that networks trained with PELU consistently improved their performance over ELU. Our results suggest that the CNNs take advantage of the added flexibility provided by learning the proper activation shape. As training progresses, we have observed that the CNNs change the parametric form of their PELU both across the layers and across the epochs. In terms of possible implications of
our results, parameterizing other activation functions could be worth investigating. Functions like Softplus, Sigmoid or Tanh may prove to be successful in some cases with proper parameterizations. Other interesting avenues for future work include applying PELU to other network architectures, such as recurrent neural networks, and to other tasks, such as object detection
ACKNOWLEDGEMENTS
We thankfully acknowledge the support of NVIDIA Corporation for providing the Tesla K80 and K20 GPUs for our experiments.

This paper proposes a modification of the ELU activation function for neural networks, by parameterizing it with 2 trainable parameters per layer. This parameter is proposed to more effectively counter vanishing gradients.
My main concern regarding this paper is related to the authors' claims about the effectiveness of PELU. The analysis in Sections 2 and 3 discusses how PELU might improve training by combating gradient propagation issues. This by itself does not imply that improved generalization will result, only that models may be easier to train. However, the experiments all seek to demonstrate improved generalization performance.
But this could in principle be due to a better inductive bias, and have nothing to do with the optimization analysis. None of the experiments are designed to directly support the stated theoretical advantage of PELU compared to ELU in optimizing models.
In the response to the pre-review question, the authors state that the claims in Section 2 and 3.3 are meant to apply to generalization performance. I fail to see how this is true for most claims, except the flexibility claim. As the authors agree, better training may or may not lead to better out-of-sample performance. I can only agree that having flexibility can sometimes help the network adapt its inductive bias to the problem (instead of overfitting), but this is a much weaker claim compared to the mathematical justifications for improved optimization.
On selection of learning hyperparameters:
The authors state in the discussion on OpenReview that the learning rates selected were favorable to ReLU, and not PELU. However, this does not guarantee that they were not unfavorable to ELU. It raises the question: can a regime be constructed where ELU has better performance than PELU? If so, how can we draw the conclusion that PELU is better?
Overall, I am not yet convinced by the experimental setup and the match between theory and experiments in this paper.