LipNet: End-to-End Sentence-level Lipreading
Lipreading is the task of decoding text from the movement of a speaker*s mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).
The authors present a well thought out and constructed system for performing lipreading. The primary novelty is the end-to-end nature of the system for lipreading, with the sentence-level prediction also differentiating this with prior work. The described neural network architecture contains convolutional and recurrent layers with a CTC sequence loss at the end, and beam search decoding with an LM is done to obtain best results. Performance is evaluated on the GRID dataset, with some saliency map and confusion matrix analysis provided as well. Overall, the work seems of high quality and clearly written with detailed explanations. The final results and analysis appear good as well. One gripe is that that the novelty lies in the choice of application domain as opposed to the methods. Lack of word-level comparisons also makes it difficult to determine the importance of using sentence-level information vs. choices in model architecture/decoding, and finally, the GRID dataset itself appears limited with the grammar and use of a n-gram dictionary. Clearly the system is well engineered and final results impress, though it*s unclear how much broader insight the results yield.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
LipNet: End-to-End Sentence-level Lipreading
Lipreading is the task of decoding text from the movement of a speaker*s mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors present a well thought out and constructed system for performing lipreading. The primary novelty is the end-to-end nature of the system for lipreading, with the sentence-level prediction also differentiating this with prior work. The described neural network architecture contains convolutional and recurrent layers with a CTC sequence loss at the end, and beam search decoding with an LM is done to obtain best results. Performance is evaluated on the GRID dataset, with some saliency map and confusion matrix analysis provided as well. Overall, the work seems of high quality and clearly written with detailed explanations. The final results and analysis appear good as well. One gripe is that that the novelty lies in the choice of application domain as opposed to the methods. Lack of word-level comparisons also makes it difficult to determine the importance of using sentence-level information vs. choices in model architecture/decoding, and finally, the GRID dataset itself appears limited with the grammar and use of a n-gram dictionary. Clearly the system is well engineered and final results impress, though it*s unclear how much broader insight the results yield.
assistant
The authors present a well thought out and constructed system for performing lipreading. The primary novelty is the end-to-end nature of the system for lipreading, with the sentence-level prediction also differentiating this with prior work. The described neural network architecture contains convolutional and recurrent layers with a CTC sequence loss at the end, and beam search decoding with an LM is done to obtain best results. Performance is evaluated on the GRID dataset, with some saliency map and confusion matrix analysis provided as well. Overall, the work seems of high quality and clearly written with detailed explanations. The final results and analysis appear good as well. One gripe is that that the novelty lies in the choice of application domain as opposed to the methods. Lack of word-level comparisons also makes it difficult to determine the importance of using sentence-level information vs. choices in model architecture/decoding, and finally, the GRID dataset itself appears limited with the grammar and use of a n-gram dictionary. Clearly the system is well engineered and final results impress, though it*s unclear how much broader insight the results yield.