A variety of learning objectives have been recently proposed for training generative models. We show that many of them, including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, VAE, β-VAE, adversarial autoencoders, AVB, and InfoVAE, are Lagrangian duals of the same primal optimization problem. This generalization reveals the implicit modeling trade-offs between flexibility and computational requirements being made by these models. Furthermore, we characterize the class of all objectives that can be optimized under certain computational constraints. Finally, we show how this new Lagrangian perspective can explain undesirable behavior of existing methods and provide new principled solutions.
Update after rebuttal
==========
Thanks for your response on my questions. The stated usefulness of the method unfortunately do not answer my worry about the significance. It remains unclear to me how much "real" difference the presented results would make to advance the existing work on generative models. Also, the authors did not promised any major changes in the final version in this direction, which is why I have reduced my score.
I do believe that this work could be useful and should be resubmitted. There are two main things to improve. First, the paper need more work on improving the clarity. Second, more work needs to be added to show that the paper will make a real difference to advance/improve existing methods.
==========
Before rebuttal
==========
This paper proposes an optimization problem whose Lagrangian duals contain many existing objective functions for generative models. Using this framework, the paper tries to generalize the optimization problems by defining computationally-tractable family which can be expressed in terms of existing objective functions. 
The paper has interesting elements and the results are original. The main issue is that the significance is unclear. The writing in Section 3 is unclear for me, which further made it challenging to understand the consequences of the theorems presented in that section. 
Here is a big-picture question that I would like to know answer for. Do the results of sec 3 help us identify a more useful/computationally tractable model than exiting approaches? Clarification on this will help me evaluate the significance of the paper.
I have three main clarification points. First, what is the importance of T1, T2, and T3 classes defined in Def. 7, i.e., why are these classes useful in solving some problems? Second, is the opposite relationship in Theorem 1, 2, and 3 true as well, e.g., is every linear combination of beta-ELBO and VMI is equivalent to a likelihood-based computable-objective of KL info-encoding family? Is the same true for other theorems?
Third, the objective of section 3 is to show that "only some choices of lambda lead to a dual with a tractable equivalent form". Could you rewrite the theorems so that they truly reflect this, rather than stating something which only indirectly imply the main claim of the paper.
Some small comments:
- Eq. 4. It might help to define MI to remind readers.
- After Eq. 7, please add a proof (may be in the Appendix). It is not that straightforward to see this. Also, I suppose you are saying Eq. 3 but with f from Eq. 4.
- Line after Eq. 8, D_i is "one" of the following... Is it always the same D_i for all i or it could be different? Make this more clear to avoid confusion.
- Last line in Para after Eq. 15, "This neutrality corresponds to the observations made in.." It might be useful to add a line explaining that particular "observation"
- Def. 7, the names did not make much sense to me. You can add a line explaining why this name is chosen.
- Def. 8, the last equation is unclear. Does the first equivalence impy the next one? 
- Writing in Sec. 3.3 can be improved. e.g., "all linear operations on log prob." is very unclear, "stated computational constraints" which constraints?