Latent Attention For If-Then Program Synthesis
Automatic translation from natural language descriptions into programs is a longstanding challenging problem. In this work, we consider a simple yet important sub-problem: translation from textual descriptions to If-Then programs. We devise a novel neural network architecture for this task which we train end-toend. Specifically, we introduce Latent Attention, which computes multiplicative weights for the words in the description in a two-stage process with the goal of better leveraging the natural language structures that indicate the relevant parts for predicting program elements. Our architecture reduces the error rate by 28.57% compared to prior art [3]. We also propose a one-shot learning scenario of If-Then program synthesis and simulate it with our existing dataset. We demonstrate a variation on the training procedure for this scenario that outperforms the original procedure, significantly closing the gap to the model trained with all data.
In this paper, the authors applied deep learning models for automatic translation of natural language descriptions into corresponding IFTTT recipes.  They investigated bi-directional LSTM and attention models.  They used one-shot learning for new channels and functions which have a very few recipes. 
The paper is not organized well, especially the last sections. There is no conclusion or discussion section. It doesn’t provide any required explanation to emphasize the contribution. Presentation of the paper is bad; model architecture and formula descriptions are not clearly presented.   The paper was not written clearly, especially from the sections 4 to 7.  There are grammatical mistakes, and it is very hard to understand. The explanation of figures and tables are not correctly represented, and the data provided in the tables is not accurately represented in the respective sections. Architecture diagram of the model is not provided for easy understanding.  Symbols and notions are not used consistently. Symbols and notions are mixed up.  Same symbols are used for different variables/parameters. Authors used non-standard terms (E.g., Depending layer, Dependent layer, etc.).   It is difficult to evaluate the originality of this work. The authors claim that there is only one existing work [14] on this problem. Novelty in the proposed models is not clearly explained.  Formulization for HA is not clear in section 4.2.2.  Moreover, it has only a small practical impact since this problem is more specific.  The experimental settings differ for each model; hence comparisons among them are not valid.  The data used for the base line model [14] and the proposed models are different. The authors have mentioned that many recipes used in [14] are no longer downloaded from IFTTT.com (section 5.1).  Moreover, the experimental settings differ for the models HA and BDLSTM.   Model evaluation methods are not described in detail. LSTM description differs from [19].  All the symbols and notations used are not explained in section 4.1. The formulations are not clear. Justifications are not provided for choosing BDLSTM.    1) In the abstract, it is mentioned as HA outperforms by 10 % but in the introduction it is mentioned as 11 points. (line no. 8 and 55) 2) Section - Introduction: Please justify why BDLSTM is chosen for comparison than the other models.  3) Please justify why 10 is chosen as number of models for average. 4) In Line no. 38, it is mentioned as BDLSTM improves over [14] by 15%. But Table 2. shows BDLSTM (average 10 models) improves over [14]. Please also check the same in the line nos. 48 to 51. 5) Table 2.  shows HA (average 10 models) improves over [14] but in line no. 52 to 55, it is mentioned just HA.  6) The purpose of one-shot learning is not mentioned. Which setting it handles well? (line no. 56-58) 7) Section 4 - Model architecture:  how variable-length descriptions are converted into a fixed-size vectors? Please explain. 8) Please check the equation of ht. The dimension of Whh ht-1 is not same as the other terms Wxh xt and bh. 9) The dimension of ht is not mentioned. 10)  All the notations (symbols) used in the equations are not described. For e.g., i, f, o, g, etc. 11)  Please use different symbol for i in wi  (line no. 139) since i is used already.   12)  Please check the equation of ct  : dot product is wrong. It should be element-wise multiplication.  13)  The symbols t and T are mixed up in the equations and throughout the section 4. Please use different symbols for Transpose, Affine transform and upper bound. 14)  Please change pointwise production to element-wise multiplication (line no. : 131) 15)  In line no. 136, order of g index is not correct. 16)  Please explain the proposed model with the architecture diagram. How the models are built is not explained in detail. 17)  How data is embedded to vector is not described?  18)  In line no. 153, please cite and specify other works. 19)  How importance of each word is measured and how weights are computed? (line no. 154). Please explain in detail. 20)  The proposed Hierarchical Attention model is not explained clearly. Please explain the novelty of the proposed model ? 21)  What are all the other “extra subtleties” does HA capture and how? (line no. 165) The authors provided only one example on “to”. 22)  Please cite and specify “the usual attention mechanism”. (line no. 166) 23)  Symbols are not consistently used for tokens: wi  (line no. 139) and wT  (line no. 173). 24)  Depending layer, Dependent layer and Output representation are not standard terminologies. 25)  The formulation in the sub-sub-section 4.2.2 is not clear. 26)  Section 5 : In line no. 186, dependent -> hierarchical 27)  Development set and Gold Test Set are not standard terminologies. 28)  Table 2: Comparison of the model [14] with HA, HA (average 10 models), BDLSTM, BDLSTM (average 10 models) is not valid since the same dataset used by [14] is not used for this work. The authors say they are not able to download many recipes (line no. 191) as used in [14].  29)  The non-frequent tokens are mapped using < UNK >. How typos are handled; which are also not in the top 4000 most frequent tokens? (line no. 202) 30)  Why the scaling factor is different for HA and BDLSTM? (line nos. 208 and 218) 31)  Why the embedding vector sizes are different? (line nos. 215 and 221) 32)  It is not clearly explained how over fitting is avoided. (line no. 209) 33)  Error in representing the data presented in Table 2. (line nos. 230-231)  34)  How will BDLSTM (average 10 models) be 10% better than any single trained model? 35)  Line no. 236 : Performance of HA is not very much greater than BDLSTM 36)  Evaluation methodologies are not described clearly. Which metrics are used for the evaluations of the models? 37)  Which mistakes do HA and BDLSTM make? Please specify (line nos. 238-239) 38)  It is mentioned the error rate is reduced by half. But it is not shown how. 39)  Section 6 : Used same symbol for top-20 set and non-top-20 set 40)  All the models mentioned in Table 4. are trained differently. Hence the comparison is not valid. 41)  Please explain why it is difficult to have balance data for BDLSTM. 42)  Table 4: Why for BDLSTM - Minor (NonTop20) for SkewTop20 is very low (4.01%)? 43)  Section 7 is incomplete. 44)  How HA matches the author’s expectations? (line no. 304)   Minor comments: 1) Long sentence line no. 44-46 2) Please change points to percentage in lines 50, 55 3) Please change percentage points to percentage 4) Typo fo - of in line nos. 72 and  137 5) article ‘the’ is repeated : line nos. 168-169 6) English is not correct (line no. 175-178) – conjunctions are missed 7) Line no. 200 : such as characters - > characters such as 8) Present tense and past tense are mixed up. (e.g., line nos. 227, 228)