Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the highdimensional latent code generated by the model, which needs to be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs), InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline while yielding comparable likelihood scores and reducing the NFEs on CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. Additional details at https://sites.google.com/view/infocnf-iclr/
This paper examines the problem of extending continuous normalizing flows (CNFs) to conditional modeling. The authors propose a model, InfoCNF, which models a latent code split into two partitions: one that is unique to a class, and another that is more general. InfoCNF relies on the accuracy of ODE solvers, so the paper also proposes a method that learns optimal error tolerances of these solvers. They perform experiments on CIFAR10, showing that InfoCNF outperforms a baseline in accuracy and negative log-likelihood. They also ablate through experiments that demonstrate the utility of learning the ODE error tolerances. 
CNFs are an exciting tool, and it's an important problem to devise methodology that applies CNFs to conditional modeling tasks. The idea of splitting latent codes into two separate components -- one supervised, the other more general -- is interesting. And the approach to learning error tolerances is a good idea.
The main drawback of this paper is the lack of clarity. It is poorly written and the presented model is not clearly motivated. Even after reading through the paper multiple times, I find it difficult to understand various aspects of InfoCNF. Below are some examples of this lack of clarity:
- When motivating Conditional CNF (CCNF), the details for training the model are unclear. What is the loss function, and how does it balance between modeling x and learning the auxiliary distribution? Although these may seem like small details, since InfoCNF builds off on CCNF, it is crucial to solidify an understanding of the training procedure and how the auxiliary task relates to modeling x. Moreover, the cited reference (Kingma and Dhariwal, 2018) does not contain these details (and there is no mention of object classification in the paper, contrary to the claim on page 3). It would be helpful to cite other references when mentioning that this approach is widely used. 
- The definition of InfoCNF is unclear. The variable y has been used to denote the image label, so why are there now L latent variables y_1, ..., y_L? The following terms of equation 4 are undefined: L_{NLL}, L_{Xent}, and y_hat. Although some readers would be able to understand these definitions from context (flow-based NLL, cross entropy loss, and the logits provided by the auxiliary distribution q), they are never explicitly defined and result in confusion and ambiguity. Most importantly, p(x|z,y) is never made explicit; although one can infer from context that  is transformed to x using CNFs, it is never made explicit in the definition (and that these flows only appear in L_{NLL}). Overall, the motivation for splitting the latent code into two pieces is not clearly explained, and the paper should spend more time arguing for this.
The paper compares InfoCNF to a single baseline (CCNF). I understand that the paper is proposing the first method that combines CNFs with conditional modeling, but there are plenty of non-CNF flow-based conditional approaches that could've been compared. The paper goes over these approaches in Section 5 and discusses their differences with InfoCNF, but these are never experimented with. It seems possible that these models could be extended in a straightforward manner to use CNFs instead of other normalizing flows. Even comparing with these models without CNFs would have been interesting. I think that using a single baseline, instead of applying a more complete set of comparisons, hurts the persuasiveness of their method.
In addition to comparing to a single baseline, the paper only compares for a single dataset (CIFAR10). A more convincing experiments section would compare against more models on more than a single dataset.
The paper proposes a promising idea to an important problem. Due to the lack of clarity throughout the paper and incomplete comparisons, I would argue to (weakly) reject.