Preconditioning an minimization algorithm improve its convergence and can lead to a minimizer in one iteration in some extreme cases. There is currently no analytical way for finding a suitable preconditioner. We present a general methodology for learning the preconditioner and show that it can lead to dramatic speed-ups over standard optimization techniques.
This paper attempts to learn a preconditioner for optimization, specifically for the Dual space preconditioned descent (DPGD). 
- The techniques used to learn the preconditioner are heuristic, not scalable and without justification or ablation studies. 
- It does not compare against "standard" optimization techniques that construct data-driven preconditioners such as Adam or Adagrad or even to more Newton, natural gradient methods that use the Hessian or the Fisher information matrix as preconditioners. It shows ad-hoc synthetic experiments in dimensions 1 and 50. This is clearly not enough. 
Detailed review below:
- Section 2: Please explain why Legendre functions are useful in ML. For assumption 1, 2; it needs to be explained why these hold for a given f*. What constraints do you need on f? What functions satisfy these? Please explain this explicitly. 
- Section 3: What is the number of points x_i needed in high dimensions to learn? Is it even possible to scale up this method to high dimensions?
- Constructing \mu requires computing the determinant of the Jacobian. What is the computational complexity? Moreover, it seems that we need access to the \nabla f(x) for all x in D(f)? 
- Please state all the assumptions in the beginning rather than introducing one at a time in the propositions. 
- Remark 1: It is unclear that the cost of an inverse Hessian matrix is more than the procedure proposed in this paper. 
- Section 3.5: Please explain what is the advantage of this learned optimizer compared to other methods? Note that there is literature on non-smooth optimization and methods like sub-gradient descent can be used in this case. 
- What is the justification for the selection of the loss function and log-rescaling?
- The result of Lemma 1 is standard. Please acknowledge this. 
-  Section 4: "The step-size is set to 1". It seems that the optimizer has been overfit and engineered to work on this specific problem. Either these decisions need to be justified, there needs to be an ablation study or there needs to be a larger set of experiments.