Published as a conference paper at ICLR 2018 BACKPROPAGATION THROUGH THE VOID: OPTIMIZING CONTROL VARIATES FOR BLACK-BOX GRADIENT ESTIMATION
Gradient-based optimization is the foundation of deep learning and reinforcement learning, but is difficult to apply when the mechanism being optimized is unknown or not differentiable. We introduce a general framework for learning low-variance, unbiased gradient estimators, applicable to black-box functions of discrete or continuous random variables. Our method uses gradients of a surrogate neural network to construct a control variate, which is optimized jointly with the original parameters. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.
This paper suggests a new approach to performing gradient descent for blackbox optimization or training discrete latent variable models. The paper gives a very clear account of existing gradient estimators and finds a way to combine them so as to construct and optimize a differentiable surrogate function. The resulting new gradient estimator is then studied both theoretically and empirically. The empirical study shows the benefits of the new estimator for training discrete variational autoencoders and for performing deep reinforcement learning.
To me, the main strengths of the paper is the very clear account of existing gradient estimators (among other things it helped me understand obscurities of the Q-prop paper) and a nice conceptual idea. The empirical study itself is more limited and the paper suffers from a few mistakes and missing information, but to me the good points are enough to warrant publication of the paper in a good conference like ICLR.
Below are my comments for the authors.
---------------------------------
General, conceptual comments:
When reading (6), it is clear that the framework performs regression of $c_\phi$ towards the unknown $f$ simultaneously with optimization over $c_\phi$.
Taking this perspective, I would be glad to see how the regression part performs with respect to standard least square regression,
i.e. just using $||f(b)-c_\phi(b)||^2$ as loss function. You may compare the speed of convergence of $c_\phi$ towards $f$ using (6) and the least squared error.
You may also investigate the role of this regression part into the global g_LAX optimization by studying the evolution of the components of (6).
Related to the above comment, in Algo. 1, you mention "f(.)" as given to the algo. Actually, the algo does not know f itself, otherwise it would not be blackbox optimization. So you may mean different things. In a batch setting, you may give a batch of [x,f(x) (,cost(x)?)] points to the algo. You more probably mean here that you have an "oracle" that, given some x, tells you f(x) on demand. But the way you are sampling x is not specified clearly.
This becomes more striking when you move to reinforcement learning problems, which is my main interest. The RL algorithm itself is not much specified. Does it use a replay buffer (probably not)? Is it on-policy or off-policy (probably on-policy)? What about the exploration policy? I want to know more... Probably you just replace (10) with (11) in A2C, but this is not clearly specified.
In Section 4, can you explain why, in the RL case, you must introduce stochasticity to the inputs? Is this related to the exploration issue (see above)?
Last sentence of conclusion: you are too allusive about the relationship between your learned control variate and the Q-function. I don't get it, and I want to know more...
-----------------------------------
Local comments:
Backpropagation through the void: I don't understand why this title. I'm not a native english speaker, I'm probably missing a reference to something, I would be glad to get it.
Figure 1 right. Caption states variance, but it is log variance. Why does it oscillate so much with RELAX?
Beginning of 3.1: you may state more clearly that optimizing $c_\phi$ the way you do it will also "minimize" the variance, and explain better why ("we require the gradient of the variance of our gradient estimator"...). It took me a while to get it.
In 3.1.1 a weighting based on $\d/\d\theta log p(b)$ => shouldn't you write $... log p(b|\theta)$ as before?
Figure 2 is mentioned in p.3, it should appear much sooner than p6.
In Figure 2, there is nothing about the REINFORCE PART. Why?
In 3.4 you alternate sums over an infinite horizon and sums over T time steps. You should stick to the T horizon case, as you mention the case T=1 later.
p6 Related work
The link to the work of Salimans 2017 is far from obvious, I would be glad to know more...
Q-prop (Haarnoja et al.,2017): this is not the adequate reference to Q-prop, it should be (Gu et al. 2016), you have it correct later ;)
Figure 3: why do you stop after so few epochs? I wondered how expensive is the computation of your estimator, but since in the RL case you go up to 50 millions (or 4 millions?), it's probably not the issue. I would be glad to see another horizontal lowest validation error for your RELAX estimator (so you need to run more epochs).
"ELBO" should be explained here (it is only explained in the appendices).
6.2, Table 1: Best obtained training objective: what does this mean? Should it be small or large? You need to explain better. How much is the modest improvement (rather give relative improvement in the text?)? To me, you should not defer Table 3 to an appendix (nor Table 4).
Figure 4: Any idea why A2C oscillates so much on inverted pendulum? Any idea why variance starts to decrease after 500 episodes using RELAX? Isn't related to the combination of regression and optimization, as suggested above?
About Double Inverted Pendulum, Appendix E3 mentions 50 million frames, but the figure shows 4 millions steps. Where is the truth?
Why do you give steps for the reward, and episodes for log-variance? The caption mentions "variance (log-scale)", but saying "log-variance" would be more adequate.
p9: the optimal control variate: what is this exactly? How do you compare a control variate over another? This may be explained in Section 2.
GAE (Kimura, 2000). I'm glad you refer to former work (there is a very annoying tendency those days to refer only to very recent papers from a small set of people who do not correctly refer themselves to previous work), but you may nevertheless refer to John Schulman's paper about GAEs anyways... ;)
Appendix E.1 could be reorganized, with a common hat and then E.1.1 for one layer model(s?) and E.1.2 for the two layer model(s?)
A sensitivity analysis wrt to your hyper-parameters would be welcome, this is true for all empirical studies.
In E2, is the output layer linear? You just say it is not ReLU...
The networks used in E2 are very small (a standard would be 300 and 400 neurons in hidden layers). Do you have a constraint on this?
"As our control variate does not have the same interpretation as the value function of A2C, it was not directly clear how to add reward bootstrapping and other variance reduction techniques common in RL into our model. We leave the task of incorporating these and other variance reduction techniques to future work."
First, this is important, so if this is true I would move this to the main text (not in appendix).
But also, it seems to me that the first sentence of E3 contradicts this, so where is the truth?
{0.01,0.003,0.001} I don't believe you just tried these values. Most probably, you played with other values before deciding to perform grid search on these, right?
The same for 25 in E3.
Globally, you experimental part is rather weak, we would expect a stronger methodology, more experiments also with more difficult benchmarks (half-cheetah and the whole gym zoo ;)), more detailed analyses of the results, but to me the value of your paper is more didactical and conceptual than experimental, which I really appreciate, so I will support your paper despite these weaknesses.
Good luck! :)
---------------------------------------
Typos:
p5
monte-carlo => Monte(-)Carlo (no - later...)
taylor => Taylor
you should always capitalize Section, equation, table, figure, appendix, ...
gradient decent => descent (twice)
p11: probabalistic
p15 ELU(Djork-... => missing space