We propose a segmental neural language model that combines the representational power of neural networks and the structure learning mechanism of Bayesian nonparametrics, and show that it learns to discover semantically meaningful units (e.g., morphemes and words) from unsegmented character sequences. The model generates text as a sequence of segments, where each segment is generated either character-by-character from a sequence model or as a single draw from a lexical memory that stores multi-character units. Its parameters are fit to maximize the marginal likelihood of the training data, summing over all segmentations of the input, and its hyperparameters are likewise set to optimize held-out marginal likelihood. To prevent the model from overusing the lexical memory, which leads to poor generalization and bad segmentation, we introduce a differentiable regularizer that penalizes based on the expected length of each segment. To our knowledge, this is the first demonstration of neural networks that have predictive distributions better than LSTM language models and also infer a segmentation into word-like units that are competitive with the best existing word discovery models.
[Note to the authors: I was assigned this paper after the reviewing deadline.]
The authors train language models on unsegmented text, simultaneously discovering word boundaries
without direct supervision. Given the past history, but ignoring past segmentation decisions
to keep computations tractable, the model predicts the next character segment (word-like unit)
by combining a character-level LSTM with a lexical memory. To prevent overusing the
lexical memory, which would lead to poor generalization, the authors propose a segment
length penalty.
Strengths:
The model architecture is interesting, combining the benefits of a character-level
model (open vocabulary) with those of a lexical model (effective for frequent character
sequences).
Despite the exponential number of possible segmentations, inference remains tractable
using dynamic programming (with some simplifying assumptions).
The ablation study clearly shows that both the lexical memory and the length penalty
contribute significantly.
Weaknesses:
The writing quality is somewhat weak. Many errors should have been caught when
proofreading the paper (e.g. "The segmentation decisions and decisions" and 
"The the characters" on page 1).
I am confused by the key-value pairs of the lexical memory. Shouldn't character
sequences be keys, and their trainable vector representations be values?
It is hard to evaluate how good the language models are, as the strength of the
baselines is unclear. How well-tuned is the LSTM?
Comparison to some other segmentation approaches (not necessarily with language modeling)
is limited. In particular, adaptor grammars perform very well on the Brent corpus [1].
However, [2] is mentioned briefly. As these other approaches work better for segmentation,
the authors should carefully justify why having a single model that does both language
modeling and word segmentation well matters. Many neural approaches have also been
suggested for Chinese word segmentation (among others [3]). In these papers, results on the
PKU dataset are much better.  Are these directly comparable with yours?
I would have liked a finer analysis of the impact of the length penalty.
A plot showing how validation likelihood and segmentation performance vary as
\lambda is increased could potentially be interesting.
[1] Johnson and Goldwater. "Improving nonparameteric Bayesian inference: experiments on
unsupervised word segmentation with adaptor grammars", HLT, 2009
[2] Berg-Kirkpatrick et al. "Painless Unsupervised Learning with Features", NAACL, 2010
[3] Yang et al. Yang, Jie, Yue Zhang, and Fei Dong. "Neural Word Segmentation with Rich Pretraining", ACL, 2017