LEARNING A GENERATIVE MODEL FOR VALIDITY IN COMPLEX DISCRETE STRUCTURES
Deep generative models have been successfully used to learn representations for high-dimensional discrete spaces by representing discrete objects as sequences and employing powerful sequence-based deep models. Unfortunately, these sequencebased models often produce invalid sequences: sequences which do not represent any underlying discrete structure; invalid sequences hinder the utility of such models. As a step towards solving this problem, we propose to learn a deep recurrent validator model, which can estimate whether a partial sequence can function as the beginning of a full, valid sequence. This validator provides insight as to how individual sequence elements influence the validity of the overall sequence, and can be used to constrain sequence based models to generate valid sequences -and thus faithfully model discrete objects. Our approach is inspired by reinforcement learning, where an oracle which can evaluate validity of complete sequences provides a sparse reward signal. We demonstrate its effectiveness as a generative model of Python 3 source code for mathematical expressions, and in improving the ability of a variational autoencoder trained on SMILES strings to decode valid molecular structures.
The authors use a recurrent neural network to build generative models of sequences in domains where the vast majority of sequences is invalid. The basic idea, outlined in Eq. 2, is moderately straightforward: at each step, use an approximation of the Q function for subsequences of the appropriate length to pick a valid extension. There are numerous details to get right. The writing is mostly clear, and the examples are moderately convincing. I wish the paper had more detailed arguments and discussions.
I question the appropriateness of Eq. 2 as a target. A correctly learned model will put positive weight on valid sequences, but it may be an arbitrarily slow way to generate diverse sequences, depending on the domain. For instance, imagine a domain of binary strings where the valid sequences are the all 1 sequence, or any sequence beginning with a 0. Half the generated sequences would be all 1's in this situation, right? And it's easy to construct further examples that are much worse than this?
The use of Bayesian active learning to generate the training set feels like an elegant idea. However, I wish there were more clarity about what was ad hoc and what wasn't. For instance, I think the use of  dropout to get q is suspect (see for instance https://arxiv.org/abs/1711.02989), and I'd prefer a little more detail on statements like "The nonlinearity of g(·) means that our Monte
Carlo approximation is biased, but still consistent." Do we have any way of quantifying the bias? Is the statement about K=16 being reasonable a statement about bias, variance, or both?
For Python strings: 
- Should we view the fact that high values of tau give a validity of 1.0 as indicative that the domain's constraints are fairly easy to learn?
- "The use of a Boltzmann policy allows us to tune the temperature parameter to identify policies
which hit high levels of accuracy for any learned Q-function approximation." This is only true to the extent the domain is sufficiently "easy" right? Is the argument that even in very hard domains, you might get this by just having an RNN which memorized a single valid sequence (assuming at least one could be found)?
- What's the best explanation for *why* the active model has much higher diversity? I understand that the active model is picking examples that tell us more about the uncertainty in w, but it's not obvious to me that means higher diversity. Do we think this is a universal property of domains?
- The highest temperature active model is exploring about half of valid sequences (modulo the non-tightness of the bound)? Have you tried gaining some insight by generating thousands of valid sequences manually and seeing which ones the model is rejecting?
- The coverage bound is used only for for Python expressions, right? Why not just randomly sample a few thousand positives and use that to get a better estimate of coverage? Since you can sample from the true positive set, it seems that your argument from the appendix about the validation set being "too similar to the training set" doesn't apply?
- It would be better to see a comparison to a strong non-NN baseline. For instance, I could easily make a PCFG over Python math expressions, and use rejection sampling to get rid of those that aren't exactly length 25, etc.?
I question how easy the Python strings example is. In particular, it might be that it's quite an easy example (compared to the SMILES) example. For SMILES, it seems like the Bayesian active learning technique is not by itself sufficient to create a good model? It is interesting that in the solubility domain the active model outperforms, but it would be nice to see more discussion / explanation.
Minor note: The incidence of valid strings in the Python expressions domain is (I believe) > 1/5000, although I guess 1 in 10,000 is still the right order of magnitude.
If I could score between "marginal accept" and "accept" I would.