DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks
In this work, we present a compact, modular framework for constructing new recurrent neural architectures. Our basic module is a new generic unit, the Transition Based Recurrent Unit (TBRU). In addition to hidden layer activations, TBRUs have discrete state dynamics that allow network connections to be built dynamically as a function of intermediate activations. By connecting multiple TBRUs, we can extend and combine commonly used architectures such as sequence-to-sequence, attention mechanisms, and recursive tree-structured models. A TBRU can also serve as both an {em encoder} for downstream tasks and as a {em decoder} for its own task simultaneously, resulting in more accurate multi-task learning. We call our approach Dynamic Recurrent Acyclic Graphical Neural Networks, or DRAGNN. We show that DRAGNN is significantly more accurate and efficient than seq2seq with attention for syntactic dependency parsing and yields more accurate multi-task learning for extractive summarization tasks.
The authors present a general framework for defining a wide variety of recurrent neural network architectures, including seq2seq models, tree-structured models, attention, and a new family of dynamically connected architectures. The framework defines a new, general-purpose recurrent unit called the TBRU, which takes a transition system, defining and constraining its inputs and outputs, and input function which defines the mapping between raw inputs and fixed-width vector representations, and recurrence function that defines the inputs to each recurrent step as a function of the current state, and an RNN cell that computes the output from the input (fixed and recurrent). Many example instantiations of this framework are provided, including sequential tagging RNNs, Google’s Parsey McParseface parser, encoder/decoder networks, tree LSTMs and less familiar examples that demonstrate the power this framework. The most interesting contribution of this work is the ease by which it can be used to incorporate dynamic recurrent connections through the definition of the transition system. In particular, this paper explores the application of these dynamic connections to syntactic dependency parsing, both as a standalone task, and by multitasking parsing with extractive summarization, using the same compositional phrase representations as features for the parser and summarization (previous work used discrete parse features), which is particularly simple/elegant in this framework. In experimental results, the authors demonstrate that such multitasking leads to more accurate summarization models, and using the framework to incorporate more structure into existing parsing models also leads to increased accuracy with no big-oh efficiency loss (compared with e.g. attention). The “raison d’etre,” in particular the example, perhaps described even more thoroughly/explicitly, should be made as clear as possible as soon as possible. This is the most important contribution, but it gets lost in the description and presentation as a framework — emphasizing that attention, seq2seq, etc can be represented in the framework is distracting and makes it seem less novel than it is. AnonReviewer6 clearly missed this point, as did I in my first pass over the paper. To get this idea across and to emphasize the benefits of this representation, I’d love to see more detailed analysis of these representations and their importance to achieving your experimental results. I think it would also be helpful to emphasize the difference between a stack LSTM and Example 6. Overall I think this paper presents a valuable contribution, though the exposition could be improved and analysis of experimental results expanded.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks
In this work, we present a compact, modular framework for constructing new recurrent neural architectures. Our basic module is a new generic unit, the Transition Based Recurrent Unit (TBRU). In addition to hidden layer activations, TBRUs have discrete state dynamics that allow network connections to be built dynamically as a function of intermediate activations. By connecting multiple TBRUs, we can extend and combine commonly used architectures such as sequence-to-sequence, attention mechanisms, and recursive tree-structured models. A TBRU can also serve as both an {em encoder} for downstream tasks and as a {em decoder} for its own task simultaneously, resulting in more accurate multi-task learning. We call our approach Dynamic Recurrent Acyclic Graphical Neural Networks, or DRAGNN. We show that DRAGNN is significantly more accurate and efficient than seq2seq with attention for syntactic dependency parsing and yields more accurate multi-task learning for extractive summarization tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors present a general framework for defining a wide variety of recurrent neural network architectures, including seq2seq models, tree-structured models, attention, and a new family of dynamically connected architectures. The framework defines a new, general-purpose recurrent unit called the TBRU, which takes a transition system, defining and constraining its inputs and outputs, and input function which defines the mapping between raw inputs and fixed-width vector representations, and recurrence function that defines the inputs to each recurrent step as a function of the current state, and an RNN cell that computes the output from the input (fixed and recurrent). Many example instantiations of this framework are provided, including sequential tagging RNNs, Google’s Parsey McParseface parser, encoder/decoder networks, tree LSTMs and less familiar examples that demonstrate the power this framework. The most interesting contribution of this work is the ease by which it can be used to incorporate dynamic recurrent connections through the definition of the transition system. In particular, this paper explores the application of these dynamic connections to syntactic dependency parsing, both as a standalone task, and by multitasking parsing with extractive summarization, using the same compositional phrase representations as features for the parser and summarization (previous work used discrete parse features), which is particularly simple/elegant in this framework. In experimental results, the authors demonstrate that such multitasking leads to more accurate summarization models, and using the framework to incorporate more structure into existing parsing models also leads to increased accuracy with no big-oh efficiency loss (compared with e.g. attention). The “raison d’etre,” in particular the example, perhaps described even more thoroughly/explicitly, should be made as clear as possible as soon as possible. This is the most important contribution, but it gets lost in the description and presentation as a framework — emphasizing that attention, seq2seq, etc can be represented in the framework is distracting and makes it seem less novel than it is. AnonReviewer6 clearly missed this point, as did I in my first pass over the paper. To get this idea across and to emphasize the benefits of this representation, I’d love to see more detailed analysis of these representations and their importance to achieving your experimental results. I think it would also be helpful to emphasize the difference between a stack LSTM and Example 6. Overall I think this paper presents a valuable contribution, though the exposition could be improved and analysis of experimental results expanded.
assistant
The authors present a general framework for defining a wide variety of recurrent neural network architectures, including seq2seq models, tree-structured models, attention, and a new family of dynamically connected architectures. The framework defines a new, general-purpose recurrent unit called the TBRU, which takes a transition system, defining and constraining its inputs and outputs, and input function which defines the mapping between raw inputs and fixed-width vector representations, and recurrence function that defines the inputs to each recurrent step as a function of the current state, and an RNN cell that computes the output from the input (fixed and recurrent). Many example instantiations of this framework are provided, including sequential tagging RNNs, Google’s Parsey McParseface parser, encoder/decoder networks, tree LSTMs and less familiar examples that demonstrate the power this framework. The most interesting contribution of this work is the ease by which it can be used to incorporate dynamic recurrent connections through the definition of the transition system. In particular, this paper explores the application of these dynamic connections to syntactic dependency parsing, both as a standalone task, and by multitasking parsing with extractive summarization, using the same compositional phrase representations as features for the parser and summarization (previous work used discrete parse features), which is particularly simple/elegant in this framework. In experimental results, the authors demonstrate that such multitasking leads to more accurate summarization models, and using the framework to incorporate more structure into existing parsing models also leads to increased accuracy with no big-oh efficiency loss (compared with e.g. attention). The “raison d’etre,” in particular the example, perhaps described even more thoroughly/explicitly, should be made as clear as possible as soon as possible. This is the most important contribution, but it gets lost in the description and presentation as a framework — emphasizing that attention, seq2seq, etc can be represented in the framework is distracting and makes it seem less novel than it is. AnonReviewer6 clearly missed this point, as did I in my first pass over the paper. To get this idea across and to emphasize the benefits of this representation, I’d love to see more detailed analysis of these representations and their importance to achieving your experimental results. I think it would also be helpful to emphasize the difference between a stack LSTM and Example 6. Overall I think this paper presents a valuable contribution, though the exposition could be improved and analysis of experimental results expanded.