PROGRAM SYNTHESIS WITH PRIORITY QUEUE TRAINING
We consider the task of program synthesis in the presence of a reward function over the output of programs, where the goal is to find programs with maximal rewards. We introduce an iterative optimization scheme, where we train an RNN on a dataset of K best programs from a priority queue of the generated programs so far. Then, we synthesize new programs and add them to the priority queue by sampling from the RNN. We benchmark our algorithm, called priority queue training (or PQT), against genetic algorithm and reinforcement learning baselines on a simple but expressive Turing complete programming language called BF. Our experimental results show that our simple PQT algorithm significantly outperforms the baselines. By adding a program length penalty to the reward function, we are able to synthesize short, human readable programs.
This paper focuses on using RNNs to generate straightline computer programs (ie. code strings) using reinforcement learning.  The basic setup assumes a setting where we do not have access to input/output samples, but instead only have access to a separate reward function for each desired program that indicates how close a predicted program is to the correct one.  This reward function is used to train a separate RNN for each desired program.
The general space of generating straight-line programs of this form has been explored before, and their main contribution is the use of a priorty queue of highest scoring programs during training.  This queue contains the highest scoring programs which have been observed at any point in the training so far, and they consider two different objectives:  (1) the standard policy-gradient objective which tries to maximize the expected reward and (2) a supervised learning objective which tries to maximize the average probability of the top-K samples.  They show that this priority queue algorithm significantly improves the stability of the resulting synthesis procedure such that when synthesis succeeds at all, it succeeds for most of the random seeds used.
This is a nice result, but I did not feel as though their algorithm was sufficently different from the algorithm used by Liang et. al. 2017.  In Liang et. al. they keep around the best observed program for each input sample.  They argue that their work is different from Liang et. al. because they show that they can learn effectively using only objective (2) while completely dropping objective (1).  However I'm quite worried these results only apply in very specific setups.  It seems that if the policy gradient objective is not used, and there are not K different programs which generate the correct output, then the Top-K objective alone will encourage the model to continue to put equal probability on the programs in the Top-K which do not generate an incorrect output.
I also found the setup itself to be poorly motivated.  I was not able to imagine a reasonable setting where we would have access to a reward function of this form without input/output examples.  The paper did not provide any such examples, and in their experiments they implement the proposed reward function by assuming access to a set of input/output examples.  I feel as though the restriction to the reward function in this case makes the problem uncessarily hard, and does not represent an important use-case.  
In addition I had the following more minor concerns:
1.  At the end of section 4.3 the paper is inconsistent about whether the test cases are randomly generated or hand picked, and whether they use 5 test cases for all problems, or sometimes up to 20 test cases.  If they are hand picked (and the number of test cases is hand chosen for each problem), then how dependant are the results on an appropriate choice of test cases?
2.  They argue that they don't need to separate train and test, but I think it is important to be sure that the generated programs work on test cases that are not a part of the reward function.  They say that "almost always" the synthesizer does not overfit, but I would have liked them to be clear about whether their reported results include any cases of overfitting (i.e. did they ensure they the final generate program always generalized)? 
3.  It is worth noting that while their technique succeeds much more consistently than the baseline genetic algorithm, the genetic algorithm actually succeeds at least once, on more tasks (19 vs. 17).  The success rate is probably a good indicator of whether the technique will scale to more complex problems, but I would have prefered to see this in the results, rather than just hoping it will be true (i.e. by including my complicated problems where the genetic algorithm never succeeds).