Connectionist Temporal Classification with Maximum Entropy Regularization
Connectionist Temporal Classification (CTC) is an objective function for end-toend sequence learning, which adopts dynamic programming algorithms to directly learn the mapping between sequences. CTC has shown promising results in many sequence learning applications including speech recognition and scene text recognition. However, CTC tends to produce highly peaky and overconfident distributions, which is a symptom of overfitting. To remedy this, we propose a regularization method based on maximum conditional entropy which penalizes peaky distributions and encourages exploration. We also introduce an entropybased pruning method to dramatically reduce the number of CTC feasible paths by ruling out unreasonable alignments. Experiments on scene text recognition show that our proposed methods consistently improve over the CTC baseline without the need to adjust training settings. Code has been made publicly available at: https://github.com/liuhu-bigeye/enctc.crnn.
• Harm the training process. The positive feedback-like error signal means CTC lacks exploration and is prone to fall into worse local minima.
• Output overconfident paths. CTC tends to concentrate all its output distribution over one specific path. On the one hand, it is not suitable to handle the situation where the segmentation boundary is ambiguous, e.g. adjacent syllables in speech recognition and action switching in continuous sign language recognition. On the other hand, the low-entropy output distribution is a symptom of overfitting [31], leading to low prediction accuracy.
• Output paths with peaky distribution. The peaky distribution is not desirable for sequence segmentation tasks when the model needs to densely predict labels for each time-step. Even if only the temporal order of labels is required, learning the correct segmentation will improve model generalization and interpretation abilities.
Motivated by the maximum entropy principle [13], we propose a maximum conditional entropy based regularization for CTC (EnCTC). We consider the conditional distribution of feasible paths given the input sequence and label sequence. EnCTC prevents the entropy of feasible paths from decreasing too fast, thus alleviating the impact of CTC's positive feedback-like error signal and encouraging exploration during training. It also prevents the probability from being dominated by a single path and solves the peaky distribution problem. It mitigates overfitting and is more suitable for depicting ambiguous segmentation boundaries.
We further consider another solution to the problem of finding the reasonable feasible path-limit the size of the feasible set. We observe that in many sequence learning tasks, the intervals between the adjacent elements in the input sequence are almost the same, e.g. characters in text recognition, syllables in speech recognition and gestures in continuous sign language recognition. We summarize this phenomenon as the equal spacing prior. Therefore, we propose an algorithm to limit the size of the CTC feasible set by eliminating these unreasonable paths that seriously violate the equal spacing prior (EsCTC). Moreover, the equal spacing prior can give theoretical explanations from the perspective of maximum entropy, which indicates that equal spacing is the best prior without any additional subjective assumptions.
The main contributions of this paper can be summarized as: (1) We propose a maximum conditional entropy regularization for CTC (EnCTC), which encourages exploration for CTC training and prevents peaky output distributions. (2) We derive from equal spacing prior a pruning algorithm (EsCTC) to effectively limit the size of CTC feasible set and give theoretical explanations from the perspective of maximum entropy. (3) We provide polynomial-time dynamic programming algorithms for calculating EnCTC, EsCTC and their combination (EnEsCTC). (4) We validate the proposed methods on scene text recognition tasks and show that these methods are able to improve the baseline model without changing training settings.
This work presents a method for end-to-end sequence learning, and more specifically in the framework of Connectionist Temporal Classification (CTC). The paper has two main contributions: - The first is a regularization of the training of the CTC objective in order to reduce the over-confidence of the model. In order to do that, the authors propose a method based on conditional entropy. More specifically, the proposed regularization would encourages the model to explore paths that are close to the dominant one.  - The second contribution is a pruning method in order to reduce the space of possible alignments. In order to do so, they suppose that the consecutive elements of a sequence have equal spacing. The authors prove that this situation corresponds to the maximum of entropy over the possible segmentations of the sequence.  The paper mainly advocates for the combination of the two contributions. The authors also provide details about dynamic programming to optimize the proposed objective. They finally compare their methods to the state-of-the-art approaches on different datasets.   The contributions of the paper are well motivated, and their effect seems to be significant according to the experimental results. The two proposed regularization are also well explained intuitively, and through supporting theory. In addition, the effect of the regularization is well analyzed in the experiments section. For all these reasons I vote for acceptance of this paper.   Minor comment: There is a typo in equation (3) in the partial derivative notation.