A Simple Fully Connected Network for Composing Word Embeddings from Characters
This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved results on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting.
 The authors propose a neural network architecture which takes the characters of a word as input along with their positions, and output a word embedding. They then use these as inputs to a GRU language model, which is evaluated on two medium size data sets made from a series of novels and the Project Gutenberg Canada books respectively. While the idea has merit, the experimental protocol is too flawed to draw any reliable conclusions. Why use Wheel of Time, which is not in the public domain, rather than e.g. text8? Why not train the model to convergence (Figure 3)? Do the learned embeddings exhibit any morphological significance, or does the model only serve a regularization purpose? As for the model itself: are the position agnostic character embeddings actually helpful in the spelling model? Does the model have the expressivity to learn the same embeddings as a look-up table? The authors are also missing a significant amount of relevant literature on the topic of building word embeddings from characters, for example: Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation, Ling et al., 2015 Enriching Word Vectors with Subword Information, Bojanowski et al. 2017 Compositional Morphology for Word Representations and Language Modelling, Botha and Blunsom 2014 Pros: - Valid idea Cons: - Too many missing references - Some modeling choices lack justification - Experiments do not provide meaningful comparisons and are not reproducible
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A Simple Fully Connected Network for Composing Word Embeddings from Characters
This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved results on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose a neural network architecture which takes the characters of a word as input along with their positions, and output a word embedding. They then use these as inputs to a GRU language model, which is evaluated on two medium size data sets made from a series of novels and the Project Gutenberg Canada books respectively. While the idea has merit, the experimental protocol is too flawed to draw any reliable conclusions. Why use Wheel of Time, which is not in the public domain, rather than e.g. text8? Why not train the model to convergence (Figure 3)? Do the learned embeddings exhibit any morphological significance, or does the model only serve a regularization purpose? As for the model itself: are the position agnostic character embeddings actually helpful in the spelling model? Does the model have the expressivity to learn the same embeddings as a look-up table? The authors are also missing a significant amount of relevant literature on the topic of building word embeddings from characters, for example: Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation, Ling et al., 2015 Enriching Word Vectors with Subword Information, Bojanowski et al. 2017 Compositional Morphology for Word Representations and Language Modelling, Botha and Blunsom 2014 Pros: - Valid idea Cons: - Too many missing references - Some modeling choices lack justification - Experiments do not provide meaningful comparisons and are not reproducible
assistant
 The authors propose a neural network architecture which takes the characters of a word as input along with their positions, and output a word embedding. They then use these as inputs to a GRU language model, which is evaluated on two medium size data sets made from a series of novels and the Project Gutenberg Canada books respectively. While the idea has merit, the experimental protocol is too flawed to draw any reliable conclusions. Why use Wheel of Time, which is not in the public domain, rather than e.g. text8? Why not train the model to convergence (Figure 3)? Do the learned embeddings exhibit any morphological significance, or does the model only serve a regularization purpose? As for the model itself: are the position agnostic character embeddings actually helpful in the spelling model? Does the model have the expressivity to learn the same embeddings as a look-up table? The authors are also missing a significant amount of relevant literature on the topic of building word embeddings from characters, for example: Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation, Ling et al., 2015 Enriching Word Vectors with Subword Information, Bojanowski et al. 2017 Compositional Morphology for Word Representations and Language Modelling, Botha and Blunsom 2014 Pros: - Valid idea Cons: - Too many missing references - Some modeling choices lack justification - Experiments do not provide meaningful comparisons and are not reproducible