We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.
After reading authors' response, I am sticking to my original decision. Authors addressed most of the issues I raised and I am happy with their response; however, I still believe the paper should not be accepted since it is not adding enough value. The problem is important and impactful. However, the algorithmic idea comes from LML (Amos 2019), and the impact on the real problems has not been demonstrated. Hence, it is adding no value algorithmically, and adding a very small value from application perspective. It is basically saying LML can be trivially applied to differentiate through CEM, and it works on some simple toy problems. To me this is mostly a sanity check. Hence, I am sticking to my weak-reject decision.
-------
The manuscript is proposing a method to make cross-entropy method (CEM) differentiable. CEM is a widely used zeroth-order optimization method. The main idea in the paper is applying the recently proposed limited multi-label projection (LML) layer in a straight-forward manner to the CEM since the major computational tool in CEM iteration is top-k selection. The authors apply the proposed method to synthetic energy-based learning and continuous control problems. 
The proposed method is definitely impactful. Considering the fact that CEM is a powerful and widely used tool, I believe the work will lead to many interesting follow-ups. In addition to these, the work is addressing computational scalability of model-based RL which is both under-explored and important problem. 
The proposed model is novel from a modelling perspective since it makes CEM part of end-to-end learnable models. Whereas, it has no algorithmic novelty since it is a straightforward application of the LML layer to the CEM problem. Lack of algorithmic novelty is not an issue but the authors should at least discuss similarities to LML (Amos 2019) in a clear manner in related work. Not including it in the related work is somewhat surprising to me.
The exposition can clearly be improved. First of all, Proposition 1 is an existing result, hence authors should give a proper citation in its definition. Second of all, Proposition 3 does not include anything about asymptotic (tau -> 0) whereas the stated one-line proof is using asymptotic arguments. Finally, there are other minor issues like Lemma1 not having a proof, proposition 1 has no statement about its proof etc.  The manuscript would significantly benefit from a thorough proof reading for mathematical completeness and correctness.
One major issue with the manuscript is the experimental study. 1) The only additional algorithmic element introduced by the manuscript is the tau and it is not experimented. Is it crucial to use the temperature parameter? If yes, what is the effect of it? Manuscript needs a collection of ablation studies discussing the tau. 2) The main claim of the paper is "...make solving the control optimization process significantly less computationally and memory expensive." This might be true but not really experimented. Authors do not report any quantitative computation time and/or memory requirement study. I believe the latent DCEM is more memory and computation efficient but quantifying this is important.
I am curious on the choice of CEM. There are other methods which can be utilized since this is basically a bi-level optimization problem. One can use implicit gradients or similar methods (like: https://arxiv.org/abs/1602.02355, https://arxiv.org/abs/1809.01465, https://arxiv.org/abs/1909.04630, http://proceedings.mlr.press/v22/domke12/domke12.pdf).  Can these methods also be utilized instead of back-propagation through optimization procedure? If yes, you should compare with them or explain why you did not. If no, you should explain why.
In summary, the paper is very impactful. On the other hand, the proposed empirical study significantly lacks in many aspects. I would be happy to increase my score if authors can address these issues.