Humans are able to both learn quickly and rapidly adapt their knowledge. One major component is the ability to incrementally combine many simple concepts to accelerates the learning process. We show that energy based models are a promising class of models towards exhibiting these properties by directly combining probability distributions. This allows us to combine an arbitrary number of different distributions in a globally coherent manner. We show this compositionality property allows us to define three basic operators, logical conjunction, disjunction, and negation, on different concepts to generate plausible naturalistic images. Furthermore, by applying these abilities, we show that we are able to extrapolate concept combinations, continually combine previously learned concepts, and infer concept properties in a compositional manner.
Update: In light of the quality of the author response, I decide to raise my score to 6: Weak Accept. I encourage the authors to keep improving their writing of section 3.4. The description of the dataset is much clearer than the previous version, but I still think it is worthwhile to use more space for a better description.
There are some other issues with this paper that I recommend the authors to address in the revisions.
- Currently all quantitative experiments are based on conjunctions. It would be nice to have quantitative results for disjunctions and negations as well to make the authors' argument more convincing.
- I would suggest removing the result of PixelCNN in the paper. Of course sampling with Langevin dynamics for PixelCNN is hard. I think a more straightforward way is to do Langevin dynamics for each conditional distribution. The current result of PixelCNN is not done in this way and can be misleading for future work.
That being said, I think the idea proposed by this paper is of novel interest, and even though the execution is not ideal, I feel the novelty itself should warrant an acceptance at ICLR.
============ Original Review ===============
This paper proposes several rules to compose different energy-based models for compositional image generation. The authors propose empirical rules for conjunction, disjunction, and negation, which in combination can be used to derive arbitrary logic expressions. The authors demonstrate that using these composition rules, energy-based models can be used for compositional image generation, continual learning, and compositional concept inference.
I agree with the authors that the composition of energy-based models is a very desirable property and a very important direction to explore, and I believe that the work of this paper can have great potential in the future. However, I feel that the current paper is not very mature, and many questions are left unanswered. Here are some detailed comments:
- What's the justification for the formula of concept disjunction? For disjunction, it makes sense to me to form a mixture of distribution where each mixture has the same weight, ie, \sum_i p(x | c_i). However, \sum_i p(x | c_i) is not proportional to \sum e^{-E(x | c_i)} because of the unknown partition functions. Therefore, when the partition functions of two energy-based models differ by a lot, the disjunction will not be a mixture with equal weights for each component. Instead, it will collapse to a single distribution where all weights are put on the component with the smallest partition function. This does not match the intuition of "disjunction".
- One way of solving the disjunction problem might be abandoning the Langevin dynamics sampling procedure (7). Sampling from a mixture of distribution is doable for arbitrary generative models as long as it is possible to sample from each component of the mixture. Why not just apply Langevin dynamics to each energy-based model in the disjunction and return the samples from each model with the same probability? This procedure cannot be trivially composed with other rules such as conjunction and negation, but it suffices to produce all results in this paper and should be more theoretically sound.
- In Table 1, it seems that energy-based models are more robust in concept inference compared to a ResNet. Why is this the case? I would envisage that the energy-based model is also a ResNet and should have the same inductive bias? Is this a property of generative classifiers? Since the results in Table 1 do not involve composition of energies it is desirable to compare the results with other generative models as well, such as conditional PixelCNN.
- Since more steps of negative sampling can give better samples and generalize better, why not use 400 steps in all experiments throughout the paper?
- In the original OpenAI paper for energy-based models [1], the authors tune the coefficients of the gradient and noise terms in Langevin dynamics for better performance. This is equivalent to doing the ordinary Langevin dynamics for a different energy function re-scaled by some temperature parameter. Did you tune the scales of Langevin dynamics as well? If so, since the energies are scaled by some temperature, the original composition rules will be problematic. Did you take account of this temperature scaling in doing the experiments? 
- There is no detailed description of model architecture or hyperparameter in the paper. The author provide no detailed information on how the model was trained, for example, the various hyperparameters in Langevin dynamics and replay buffer. No code is available. There is even no appendix. The paper is very hard to reproduce, which hurts the reliability of the experimental results.
- Writing can be improved.  The description of the dataset in section 3.4 is barely readable. The second paragraph in section 3.4 needs to be paraphrased or significantly expanded. At least there should be some more detailed description of the settings in the appendix.
- At the end of the day, why energy-based models? I can imagine that an autoregressive model can be perfectly used for all tasks described in this paper. It does not even have the problem I mentioned in disjunction, since all densities are normalized. No Langevin dynamics sampling is needed. Since autoregressive models decompose the density using the chain rule, all compositions of the densities can also nicely decompose, and sequential sampling still works.
References:
[1] Du, Yilun, and Igor Mordatch. "Implicit generation and generalization in energy-based models." arXiv preprint arXiv:1903.08689 (2019).