Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training  a semi-supervised technique for incorporating unlabeled data  sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection. Speech introduces challenges that do not appear in written text, such as the presence of disfluencies. Disfluency refers to any interruptions in the normal flow of speech, including false starts, corrections, repetitions and filled pauses. Shriberg (1994) defines three distinct parts of a speech disfluency, referred to as the reparandum , the interregnum and the repair . As illustrated in the example below, the reparandum The first kind of invasion of is the part of the utterance that is replaced or repaired, the interregnum uh I mean (which consists of a filled pause uh and a discourse marker I mean ) is an optional part of the disfluency, and the repair the first type of privacy replaces the reparandum. The fluent version is obtained by removing the reparandum and the interregnum. This paper will focus on joint disfluency detection and constituency parsing of transcribed speech. In the Switchboard treebank corpus (God-frey and Holliman, 1993; Marcus et al., 1999), which is a standard corpus for parsing studies on conversational speech, the reparanda , filled pauses and discourse markers are dominated by EDITED, INTJ and PRN nodes, respectively (see Figure 1). Filled pauses and discourse markers belong to a finite set of words and phrases, so INTJ and PRN nodes are trivial to detect (Johnson and Charniak, 2004). Detecting EDITED nodes, however, is challenging and is the main focus of disfluency detection models. Jamshid Lou et al. (2019) showed that a self-attentive constituency parser achieves state-of-the-art results for joint parsing and disfluency detection. They observed that because the Switchboard trees include both syntactic constituency nodes and EDITED nodes that indicate disfluency, training a parser to predict the Switchboard trees can be regarded as multi-task learning (where the tasks are syntactic parsing and identifying disfluencies). In this paper, we extend the multi-task learning in Jamshid Lou et al. (2019) to explore the impact of self-training (McClosky et al., 2006) and ensembling (Kitaev et al., 2019) on the performance of the self-attentive parser. We aim to answer two questions about the state-of-the-art self-attentive parser:  Does self-training improve the performance of the self-attentive parser on disfluency detection? Self-training is a semi-supervised technique for incorporating unlabeled data into a new model, where an existing model trained on manually labeled (i.e. gold) data is used to label unlabeled data. The automatically (i.e. silver) labeled data are treated as truth and combined with the gold labeled data to re-train a new model (McClosky et al., 2006; Choe and Charniak, 2016). Since neural models use rich representations of language pre-trained on a large amount of unlabeled data (Peters et al., 2018; Devlin et al., 2019), we might expect that self-training adds no new information to the self-attentive parser. Surprisingly, however, we find that self-training improves disfluency detection f-score of the BERT-based self-attentive parser , demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized embeddings. Does ensembling improve disfluency detection in speech transcripts? Ensembling is a commonly used technique for improving parsing where scores of multiple instances of the same model trained on the same or different data are combined at inference time (Dyer et al., 2016; Fried et al., 2017; Kitaev et al., 2019). We expect ensembling parsers to improve the performance of the model on disfluency detection, too. We show ensembling four self-trained parsers (using different BERT word representations) via averaging their span label scores increases disfluency detection f-score in comparison with a single self-trained parser. Parsing speech transcripts is challenging for conventional syntactic parsers, mainly due to the presence of disfluencies. In disfluent sentences, the relation between reparandum and repair is different from other words in the sentence. The repair is usually a rough copy of the reparandum, using the same or similar words in roughly the same word order 1 (Charniak and Johnson, 2001). Designed to capture tree-like structures, conventional syntactic parsers fail to detect rough copies which are strong indicators of disfluency. Moreover, the reparandum and repair often do not form a syntactic phrase, which makes detecting the reparandum even harder. For these reasons, specialized disfluency detection models were developed to remove disfluencies prior to parsing (Char-niak and Johnson, 2001; Kahn et al., 2005; Lease and Johnson, 2006) or special mechanisms were added to parsers to handle disfluencies (Rasooli and Tetreault, 2013; Honnibal and Johnson, 2014; Yoshikawa et al., 2016). Conventional parsing based models can use the syntactic location of the disfluency as a feature in a reranker (Johnson et al., 2004). A similar gain can be achieved in neural models by training a joint parsing and disfluency detection model. In this multi-task learning setting, syntactic information helps the neural model detect disfluencies more accurately (Jamshid Lou et al., 2019). State-of-the-art results for disfluency detection have been reported for Transformer models using contextualized embeddings (e.g. ELMo and BERT) (Jamshid Lou et al., 2019; Tran et al., 2019; Dong et al., 2019). The self-attention mechanism of the Transformer is apparently effective for capturing rough copy dependencies between words. A recent study shows that prosody slightly improves the parsing performance of the self-attentive model over the text-only model, especially in long sentences (Tran et al., 2019). In this paper, we use a self-attentive model for joint disfluency detection and constituency parsing. Disfluency detection models are usually trained and evaluated on the Switchboard corpus. Switchboard is the largest disfluency annotated dataset. However, only 5.9% of the words in the Switchboard are disfluent (Charniak and Johnson, 2001). 1 For example in Figure 1, the reparandum The first kind of invasion of and the repair the first type of privacy are rough copies of each other. To mitigate the scarcity of labeled data, some studies have leveraged additional data by using: (i) contextualized embeddings pre-trained on enormous amount of unlabeled data (Jamshid Lou et al., 2019; Tran et al., 2019; Bach and Huang, 2019) and (ii) synthetic data generated by adding noise in the form of disfluencies to fluent sentences (e.g. repeating, deleting or inserting words in a sentence) (Wang et al., 2018; Bach and Huang, 2019; Dong et al., 2019). By contrast, this paper focuses on self-training, which is a simple semi-supervised technique that has been effective in different NLP tasks, including parsing (McClosky et al., 2006; Clark et al., 2018; Droganova et al., 2018). To our best knowledge, this is the first work that investigates self-training a neural disfluency detection model. Another technique commonly used for improving parsing is ensembling. Ensembling is a model combination method, where scores of multiple models (they can be the same or different models, trained on the same or different data, with different random initializations) are combined in some way (Dyer et al., 2016; Choe and Charniak, 2016; Fried et al., 2017). The state-of-the-art for parsing written text is an ensemble of four BERT-based self-attentive parsers, where the parsers are combined by averaging their span label scores (Kitaev et al., 2019). While ensembling is widely used in parsing, it has not been investigated for disfluency detection. In this paper, we also explore the impact of ensembling several parsing based disfluency detection models on disfluency detection performance. Following Jamshid Lou et al. (2019), we use a self-attentive constituency parser for joint disfluency detection and syntactic parsing 2 . The parsing model is based on the architecture introduced by Kitaev and Klein (2018), which is state-of-the-art for (i) parsing written texts (Kitaev et al., 2019; Fried et al., 2019), (ii) parsing transcribed speech (Tran et al., 2019), and (iii) joint parsing and disfluency detection (Jamshid Lou et al., 2019). The self-attentive parser assigns a score s ( T ) to each tree T by calculating the sum of the potentials 2 The code is available at: https://github.com/pariajm/joint-disfluency-detection-and-parsing on its labeled constituent spans: s ( T ) = (cid:88) ( i,j,l )  T s ( i, j, l ) (1) where s ( i, j, l ) is the score of a constituent beginning at string position i ending at position j with label l . The input to the parser is a sequence of vectors corresponding to the sequence of words in a sentence followed by one or more self-attention layers. For each span ( i, j ) , a hidden vector h ij is constructed by subtracting the representations of the start and end of the span. A span classifier, including two fully connected layers followed by a non-linearity, assigns labeling scores s ( i, j, . ) to each span. Then, the highest scoring parse tree is found for a given sentence as follows:  T = argmax T s ( T ) (2) using a modified CYK algorithm. The parser introduced in Kitaev and Klein (2018) relies on an external POS tagger to predict preterminal labels, but because the parser's accuracy does not decrease when no external POS tagger is used, we use their parser here without an external POS tagger (hence, all the preterminal labels are UNK). For more details, see Kitaev and Klein (2018). We incorporate BERT (Devlin et al., 2019) in our self-attentive parser by fine-tuning the parameters as part of the training process. Following Kitaev et al. (2019), we apply a learned projection matrix on the output of BERT to project the vectors to our desired dimensionality. The representations are then fed into the parser. BERT learns the representations for sub-word units, so to extract the word representations, we consider the representations of the last sub-word unit for each word in the sentence (Kitaev et al., 2019). We train the self-attentive parser on the Penn Treebank-3 Switchboard corpus which contains gold disfluency labeled parse trees (Godfrey and Holliman, 1993; Marcus et al., 1999). Using the trained model, we parse unlabeled data and add the silver parse trees to the gold Switchboard training data and re-train the self-attentive parser using the enlarged training set. The unlabeled data we use include Fisher Speech Transcripts Part 1 (Cieri et al., 2004) and Part 2 (Cieri et al., 2005). Following Charniak and Johnson (2001), we split the Switchboard into training, dev and test sets as follows: training data consists of the sw[23] .mrg files, dev data consists of the sw4[5-9] .mrg files and test data consists of the sw4[0-1] .mrg files. All partial words 3 and punctuations are removed from the data, as they are not available in realistic ASR applications (Johnson and Charniak, 2004). Our baseline is the self-attentive parser trained on the gold Switchboard corpus with BERT word representations. The BERT-based parser is the current state-of-the-art, providing a very strong baseline for our work. We trained different versions of the baseline parser using four different BERT models, namely BERTBASE [cased | uncased] and BERTLARGE [cased | uncased] , and then selected the best model i.e. BERTBASE [cased] on the Switchboard dev set. We also tuned the hyperparameters by optimizing for performance on parsing EDITED nodes F ( SE ) . Preliminary experiments on the Switchboard dev set showed that the hyperparameters given by Kitaev et al. (2019) perform well; therefore, this is what we used here. Since random seeds lead to different results, in this paper we report average scores across 5 runs of each model initialized with different random seeds. We evaluate the self-attentive parser in terms of parsing accuracy, as well as disfluency detection. Since certain words are identified as EDITED in the parse tree, we can measure how well a parser 3 Words tagged as XX or words ending in classifies words as EDITED. We can also evaluate how accurately the parser can identify all disfluency words, i.e., the words dominated by EDITED, INTJ or PRN nodes. Therefore, we report precision (P), recall (R) and f-score (F) for both constituent spans (S) and word positions (W), where each word position is treated as labeled by all the constituents containing that word. We also report the result for subsets of constituent spans and word positions: (i) SE , the set of constituent spans labeled EDITED, (ii) WE , the set of word positions dominated by one or more EDITED nodes, and (iii) WEIP , the set of word positions dominated by one or more EDITED, INTJ or PRN nodes. For more details, see Jamshid Lou et al. (2019). To find the optimal proportion of additional silver training data, we select n percent (ranging from 10% to 90% ) of the training data in each mini-batch from silver parse trees and the rest from the gold ones. This has the same effect as re-weighting the main gold corpus as in McClosky et al. (2006). The results for using different proportions of the silver parse trees are presented in Figure 2. The BERT-based parser self-trained with 40% silver Fisher trees and 60% gold Switchboard trees is our best model. In other words, for a batch size of 30, in each mini-batch 12 parse trees come from the silver Fisher data and 18 parse trees from the gold Switchboard. All self-training results in this paper use this proportion of gold and silver parse trees. Tables 2 and 3 compare the baseline and the self-trained parser in terms of parsing and disfluency detection. The parser self-trained on the silver Fisher data increases parsing and disfluency detection performance, indicating the BERT-based model benefits from additional silver labeled data. Self-training is especially effective for recognizing EDITED disfluency nodes ( 1 . 5% increase in f-score). Only 5 . 9% of the words in the Switchboard are disfluent, and BERT is only trained on fluent texts such as books and Wikipedia, so the baseline parser may be starved of disfluent training examples. As a result, self-training on a corpus of conversational speech may compensate for the scarcity of disfluent gold data. To explore this, we tried self-training on a wide variety of fluent clean datasets, including Gigaword 5 (which is an unlabelled newswire corpus) and WSJ and Brown (which include gold parse trees of written text), but the performance did not improve significantly. This suggests that the parser benefits more from additional in-domain (i.e. conversational) silver data than additional out-of-domain (i.e. written) silver/gold data. Moreover, if we learn the embeddings as part of training instead of using pre-trained BERT, EDITED word f-score would drop from 90.9% to 86.4% and self-training on Fisher leads to little improvement (0.2% increase in EDITED word f-score compared to 1.5% improvement when using BERT). This suggests that self-training works well when the baseline model is powerful enough to predict accurate silver labels. To further investigate the influence of self-training on disfluency detection, we randomly select 100 sentences containing disfluencies from Disfluency F(W E ) F(W EIP ) Baseline 90 . the Switchboard dev set. We categorize disfluencies into repetition , correction and restart according to Shriberg's (1994) typology of speech repairs. Repetitions are repairs where the reparandum and repair portions of the disfluency are identical, while corrections are where the reparandum and repairs differ (which are much harder to de-tect). Restarts are where the speaker abandons a sentence and starts a new one (i.e. the repair is empty). As Table 4 shows, the self-trained parser outperforms the baseline in detecting all types of disfluency. It especially has a better performance at detecting corrections and restarts which are more challenging types of disfluency in comparison with repetitions. We investigate the impact of ensembling on the performance of the self-attentive parser, where we combine parsers by averaging their span label scores as follows: s ensemble ( i, j, l ) = 1 4 4 (cid:88) n =1 s n ( i, j, l ) (3) We tried different ensembling of parsers and the best result was achieved when we trained the baseline parser four times using four BERT word rep-# Model EDITED Disfluency Labels 1 Gold if if you call the any eight hundred number if you you can call up any eight hundred number Baseline if if you call the any eight hundred number if you you can call up any eight hundred number Self-trained if if you call the any eight hundred number if you you can call up any eight hundred number 2 Gold she was going to get picked up she was going to pick him up because she only    Baseline she was going to get picked up she was going to pick him up because she only    Self-trained she was going to get picked up she was going to pick him up because she only    3 Gold It goes back to you know what right what can society impose on people Baseline It goes back to you know what right what can society impose on people Self-trained It goes back to you know what right what can society impose on people 4 Gold and the money they do have they're not they do not use it wisely Baseline and the money they do have they're not they do not use it wisely Self-trained and the money they do have they're not they do not use it wisely 5 Gold For two years we didn't and we which was a kind of stupid Baseline For two years we didn't and we which was a kind of stupid Self-trained For two years we didn't and we which was a kind of stupid 6 Gold We we couldn't survive in a in a juror in a trial system without a jury Baseline We we couldn't survive in a in a juror in a trial system without a jury Self-trained We we couldn't survive in a in a juror in a trial system without a jury 7 Gold    I think it's like ninety-nine point ninety-nine think it is Baseline    I think it's like ninety-nine point ninety-nine think it is Self-trained    I think it's like ninety-nine point ninety-nine think it is 8 Gold Do you think for a big or a little place Baseline Do you think for a big or a little place Self-trained Do you think for a big or a little place Table 5: Some examples from the Switchboard dev set and corresponding EDITED disfluency labels given by the baseline and the best self-trained parser, as well as the gold (i.e. correct) labels. resentations, namely BERTBASE [cased | uncased] and BERTLARGE [cased | uncased] , and combined the results at inference time (Kitaev et al., 2019). The ensembled models not only reflect variations of different pre-trained representations but also the randomness in initialization of the models. As shown in Table 6, ensembling and self-training both improve the performance of the baseline single model on parsing and detecting EDITED disfluency nodes. Self-training is more effective than ensembling, especially for EDITED node detection. The best results are reported for ensembling the best of the self-trained parsers for each of different BERT models from the 5 random restarts 4 . We compare the performance of our best model with previous work on the Switchboard test set. As demonstrated in Table 7, our model outperforms prior work in parsing. The parsing result for our model is higher than Tran et al. (2018) which utilizes prosodic cues, as well as text based features. We compare the performance of the self-attentive parser with state-of-the-art disfluency detection models. As shown in Table 8, our model has the best f-score. We also compare our model with prior work that reported EDITED, INTJ and PRN word f-score for disfluency detection and find that our model has the best performance (see Table 9). Compared to Wang et al. (2018) which uses GANs to leverage additional unlabelled data and Bach and Huang (2019) which leverages synthetic data, our model significantly improves the recall. This demonstrates that standard techniques such as self-training and ensembling are as good or better than these specialized, complex approaches. We conduct a qualitative analysis on the Switchboard dev set to characterize the disfluencies that the baseline model cannot detect but the self-trained one can. We provide representative examples in Table 5. In general, the self-trained model is better at detecting long complex corrections (# 1-4), restarts (# 5) and stutter-like repetitions (# 6). It also does a better job of discriminating fluent repetitions and fluent parallel structures from repetition and correction types of disfluency (# 7 and 8). Figure 3 depicts a sentence parsed by the baseline and the self-trained self-attentive parser, where the self-trained model correctly predicts all disfluency EDITED nodes. As explained in Section 3, we do not use an external POS tagger, so POS tags are not available when parsing from raw text. That's why all preterminal labels in Figure 3 are shown by a dummy token i.e. UNK. We introduced a new state-of-the-art for joint disfluency detection and constituency parsing of transcribed speech. We showed that self-training and ensembling are effective methods for improving disfluency detection. A qualitative analysis of the results also indicated that self-training is helpful for detecting complicated types of disfluencies, including corrections and restarts. In future work, we intend to explore the idea of self-training for parsing written texts. We also aim at integrating syntactic parsing and self-training more closely with automatic speech recognition. The first step is to develop parsing models that parse ASR output, rather than speech transcripts. We would like to thank the anonymous reviewers for their insightful comments and suggestions. This research was supported by a Google award through the Natural Language Understanding Focused Program, by a CSIRO's DATA61 Top-up Scholarship, and under the Australian Research Councils Discovery Projects funding scheme (project number DP160102156).
abstain abstain objective result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective objective objective abstain objective objective objective result result other other other other other other other other other other other other method other other other other other other other abstain objective other other other other objective method abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain objective objective abstain other other