Recurrent Hierarchical Topic-Guided Neural Language Models
To simultaneously capture syntax and semantics from a text corpus, we propose a new larger-context language model that extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-of-the-art larger-context language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.
This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics. Overall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. Some areas for improvement: It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that. I don*t know that it makes sense to highlight as the contribution of this model that it can *simultaneously capture syntax and semantics*. It*s not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level. If this is correct, that should be expressed more precisely. It*s not clear to me what we learn from Figure 3. The claim is that *the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies*, but looking at the higher layers I*m not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Recurrent Hierarchical Topic-Guided Neural Language Models
To simultaneously capture syntax and semantics from a text corpus, we propose a new larger-context language model that extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-of-the-art larger-context language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics. Overall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. Some areas for improvement: It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that. I don*t know that it makes sense to highlight as the contribution of this model that it can *simultaneously capture syntax and semantics*. It*s not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level. If this is correct, that should be expressed more precisely. It*s not clear to me what we learn from Figure 3. The claim is that *the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies*, but looking at the higher layers I*m not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.
assistant
This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics. Overall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. Some areas for improvement: It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that. I don*t know that it makes sense to highlight as the contribution of this model that it can *simultaneously capture syntax and semantics*. It*s not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level. If this is correct, that should be expressed more precisely. It*s not clear to me what we learn from Figure 3. The claim is that *the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies*, but looking at the higher layers I*m not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.