Deep Learning with Dynamic Computation Graphs
Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.
The paper describes a novel technique to improve the efficiency of computation graphs in deep learning frameworks. An impressive speedup can be observed in their implementation within TensorFlow. The content is presented with sufficient clarity, although some more graphical illustrations could be useful. This work is relevant in order to achieve highest performance in neural network training. Pros: - significant speed improvements through dynamic batching - source code provided Cons: - the effect on a large real-world (ASR, SMT) would allow the reader to put the improvements better into context - presentation/vizualisation can be improved
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Learning with Dynamic Computation Graphs
Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper describes a novel technique to improve the efficiency of computation graphs in deep learning frameworks. An impressive speedup can be observed in their implementation within TensorFlow. The content is presented with sufficient clarity, although some more graphical illustrations could be useful. This work is relevant in order to achieve highest performance in neural network training. Pros: - significant speed improvements through dynamic batching - source code provided Cons: - the effect on a large real-world (ASR, SMT) would allow the reader to put the improvements better into context - presentation/vizualisation can be improved
assistant
The paper describes a novel technique to improve the efficiency of computation graphs in deep learning frameworks. An impressive speedup can be observed in their implementation within TensorFlow. The content is presented with sufficient clarity, although some more graphical illustrations could be useful. This work is relevant in order to achieve highest performance in neural network training. Pros: - significant speed improvements through dynamic batching - source code provided Cons: - the effect on a large real-world (ASR, SMT) would allow the reader to put the improvements better into context - presentation/vizualisation can be improved