End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, suggesting that the compositionality of cascaded systems simplifies learning and enables sophisticated search capabilities. In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall performance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermediates from a speech recognition sub-task. The model demonstrates the aforementioned benefits and outperforms the previous state-of-the-art by around +6 and +3 BLEU on the two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the English-German and English-French test sets of MuST-C. 1 1 Introduction The principle of compositionality loosely states that a complex whole is composed of its parts and the rules by which those parts are combined (Lake and Baroni, 2018). This principle is present in engineering, where task decomposition of a complex system is required to assess and optimize task allocations (Levis et al., 1994), and in natural language, where paragraph coherence and discourse analysis rely on decomposition into sentences (Johnson, 1992; Kuo, 1995) and sentence level semantics relies on decomposition into lexical units (Liu et al., 2020b). 1 All code and models are released as part of the ESPnet toolkit: https://github.com/espnet/espnet . Similarly, many sequence-to-sequence tasks that convert one sequence into another (Sutskever et al., 2014) can be decomposed to simpler sequence subtasks in order to reduce the overall complexity. For example, speech translation systems, which seek to process speech in one language and output text in another language, can be naturally decomposed into the transcription of source language audio through automatic speech recognition (ASR) and translation into the target language through machine translation (MT). Such cascaded approaches have been widely used to build practical systems for a variety of sequence tasks like hybrid ASR (Hinton et al., 2012), phrase-based MT (Koehn et al., 2007), and cascaded ASR-MT systems for speech translation (ST) (Pham et al., 2019). End-to-end sequence models like encoder-decoder models (Bahdanau et al., 2015; Vaswani et al., 2017), are attractive in part due to their simplistic design and the reduced need for hand-crafted features. However, studies have shown mixed results compared to cascaded models particularly for complex sequence tasks like speech translation (In-aguma et al., 2020) and spoken language understanding (Coucke et al., 2018). Although direct target sequence prediction avoids the issue of error propagation from one system to another in cascaded approaches (Tzoukermann and Miller, 2018), there are many attractive properties of cascaded systems, missing in end-to-end approaches, that are useful in complex sequence tasks. In particular, we are interested in (1) the strong search capabilities of the cascaded systems that compose the final task output from individual system predictions (Mohri et al., 2002; Kumar et al., 2006; Beck et al., 2019), (2) the ability to incorporate external models to re-score each individual system (Och and Ney, 2002; Huang and Chiang, 2007), (3) the ability to easily adapt individual components towards out-of-domain data (Koehn and Schroeder, 2007; Peddinti et al., 2015), and finally (4) the ability to monitor performance of the individual systems towards the decomposed sub-task (Tillmann and Ney, 2003; Meyer et al., 2016). In this paper, we seek to incorporate these properties of cascaded systems into end-to-end sequence models. We first propose a generic framework to learn searchable hidden intermediates using an auto-regressive encoder-decoder model for any decomposable sequence task (3). We then apply this approach to speech translation, where the intermediate stage is the output of ASR, by passing continuous hidden representations of discrete transcript sequences from the ASR sub-net decoder to the MT sub-net encoder. By doing so, we gain the ability to use beam search with optional external model re-scoring on the hidden intermediates, while maintaining end-to-end differentiability. Next, we suggest mitigation strategies for the error propagation issues inherited from decomposition. We show the efficacy of searchable intermediate representations in our proposed model, called the Multi-Decoder, on speech translation with a 5.4 and 2.8 BLEU score improvement over the previous state-of-the-arts for Fisher and CallHome test sets respectively (6). We extend these improvements by an average of 0.5 BLEU score through the aforementioned benefit of re-scoring the intermediate search with external models trained on the same dataset. We also show a method for monitoring sub-net performance using oracle intermediates that are void of search errors (6.1). Finally, we show how these models can adapt to out-of-domain speech translation datasets, how our approach can be generalized to other sequence tasks like speech recognition, and how the benefits of decomposition persist even for larger corpora like MuST-C (6.2). The probabilistic space of a sequence is combinatorial in nature, such that a sentence of L words from a fixed vocabulary V would have an output space S of size |V| L . In order to deal with this combinatorial output space, an output sentence is decomposed into labeled target tokens, y = ( y 1 , y 2 , . . . , y L ) , where y l  V . sequence-to-sequence tasks to learn next word prediction, which outputs a distribution over the next target token y l given the previous tokens y 1: l (cid:57) 1 and the input sequence x = ( x 1 , x t , . . . , x T ) , where T is the input sequence length. In the next sub-section we detail the training and inference of these models. Training: In an auto-regressive encoder-decoder model, the ENCODER maps the input sequence x to a sequence of continuous hidden representations h E = ( h E 1 , h Et , . . . , h ET ) , where h Et  R d . The DECODER then auto-regressively maps h E and the preceding ground-truth output tokens,  y 1: l (cid:57) 1 , to h Dl , where h Dl  R d . The sequence of decoder hidden representations form h D = ( h D 1 , h Dl , . . . , h DL ) and the likelihood of each output token y l is given by SOFTMAXOUT , which denotes an affine projection of h Dl to V followed by a softmax function. h E = ENCODER ( x ) h Dl = DECODER ( h E ,  y 1: l (cid:57) 1 ) (1) P ( y l |  y 1: l (cid:57) 1 , h E ) = SOFTMAXOUT ( h Dl ) (2) During training, the DECODER performs token classification for next word prediction by considering only the ground truth sequences for previous tokens y . We refer to this h D as oracle decoder representations, which will be discussed later. Inference: During inference, we can maximize the likelihood of the entire sequence from the output space S by composing the conditional probabilities of each step for the L tokens in the sequence. h Dl = DECODER ( h E , y 1: l (cid:57) 1 ) (3) P ( y l | x , y 1: l (cid:57) 1 ) = SOFTMAXOUT ( h Dl ) y = argmax y S L (cid:89) i =1 P ( y i | x , y 1: i (cid:57) 1 ) (4) This is an intractable search problem and it can be approximated by either greedily choosing argmax at each step or using a search algorithm like beam search to approximate y . Beam search (Reddy, 1988) generates candidates at each step and prunes the search space to a tractable beam size of B most likely sequences. As B   , the beam search result would be equivalent to equation 4. In approximate search for auto-regressive models, like beam search, the DECODER receives alternate candidates of previous tokens to find candidates with a higher likelihood as an overall sequence. This also allows for the use of external models like Language Models (LM) or Connectionist Temporal Classification Models (CTC) for re-scoring candidates (Hori et al., 2017). In this section, we present a general framework to exploit natural decompositions in sequence tasks which seek to predict some output C from an input sequence A . If there is an intermediate sequence B for which A  B sequence transduction followed by B  C prediction achieves the original task, then the original A  C task is decomposable. In other words, if we can learn P ( B | A ) then we can learn the overall task of P ( C | A ) through max B ( P ( C | A , B ) P ( B | A )) , approximated using Viterbi search. We define a first encoder-decoder SUB AB NET to map an input sequence A to a sequence of decoder hidden states, h DB . Then we define a subsequent SUB BC NET to map h DB to the final probabilistic output space of C . Therefore, we call h DB hidden intermediates . The following equations shows the two sub-networks of our framework, SUB AB NET and SUB BC NET , which can be trained end-to-end while also exploiting compositionality in sequence tasks. 2 2 Note that this framework does not use locally-normalized softmax distributions but rather the hidden representations, thereby avoiding label bias issues when combining multiple sub-systems (Bottou et al., 1997; Wiseman and Rush, 2016). h E = ENCODERA ( A )  h DB l = DECODERB ( h E , y B 1: l (cid:57) 1 ) P ( y B l | y B 1: l (cid:57) 1 , h E ) = SOFTMAXOUT ( h DB l ) (5) SUB BC NET : Note that the final prediction, given by equation 6, does not need to be a sequence and can be a categorical class like in spoken language understanding tasks. Next we will show how the hidden intermediates become searchable during inference. As stated in section 2.2, approximate search algorithms maximize the likelihood, P ( y | x ) , of the entire sequence by considering different candidates y l at each step. Candidate-based search, particularly in auto-regressive encoder-decoder models, also affects the decoder hidden representation, h D , as these are directly dependent on the previous candidate (refer to equations 1 and 3). This implies that by searching for better approximations of the previous predicted tokens, y l (cid:57) 1 = ( y BEAM ) l (cid:57) 1 , we also improve the decoder hidden representations for the next token, h Dl = ( h DBEAM ) l . As y BEAM  y , the decoder hidden representations tend to the oracle decoder representations that have only errors from next word prediction, h DBEAM  h D . A perfect search is analogous to choosing the ground truth  y at each step, which would yield h D . We apply this beam search of hidden intermediates, thereby approximating h DB with h DBBEAM . This process is illustrated in algorithm 1, which shows beam search for h DBBEAM that are subsequently passed to the SUB BC NET . 3 In line 7, we show how an external model like an LM or a CTC model can be used to generate an alternate sequence likelihood, PEXT ( y B l ) , which can be combined with the SUB AB NET likelihood, PB ( y B l | x ) , with a tunable parameter  . Algorithm 1 Beam Search for Hidden Intermediates: We perform beam search to approximate the most likely sequence for the sub-task A  B , y BBEAM , while collecting the corresponding DECODERB hidden representations, h DBBEAM . The output h DBBEAM , is passed to the final sub-network to predict final output C and y BBEAM is used for monitoring performance on predicting B . 1: Initialize: BEAM  {sos}; k  beam size; 2: h EA  ENCODERA ( x ) 3: for l = 1 to max STEPS do 4: for y B l (cid:57) 1  BEAM do 5: h DB l  DECODERB ( h EA , y B l (cid:57) 1 ) 6: for y B l  y B l (cid:57) 1 + {V} do 7: s l  P AB ( y B l | x ) 1 (cid:57)  PEXT ( y B l )  8: H  ( s l , y B l , h DB l ) 9: end for 10: end for 11: BEAM  arg k max( H ) 12: end for 13: ( s B , y BBEAM , h DBBEAM )  argmax( BEAM ) 14: Return y BBEAM  SUB AB NET Monitoring 15: Return h DBBEAM  Final SUB BC NET We can monitor the performance of the SUB AB NET by comparing the decoded intermediate sequence y BBEAM to the ground truth y B . We can also monitor the SUB BC NET performance by using the aforementioned oracle representations of the intermediates,  h DB , which can be obtained by feeding the ground truth y B to DECODERB . By passing h DB to SUB BC NET , we can observe its performance in a vacuum, i.e. void of search errors in the hidden intermediates. In order to show the applicability of our end-to-end framework we propose our Multi-Decoder model for speech translation. This model predicts a sequence of text translations y ST from an input se-3 The algorithm shown only considers a single top approximation of the search; however, with added time-complexity, the final task prediction improves with the n-best h DBBEAM for selecting the best resultant C . quence of speech x and uses a sequence of text transcriptions y ASR as an intermediate. In this case, the SUB AB NET in equation 5 is specified as the ASR sub-net and the SUB BC NET in equation 6 is specified as the MT sub-net. Since the MT sub-net is also a sequence prediction task, both sub-nets are encoder-decoder models in our architecture (Bah-danau et al., 2015; Vaswani et al., 2017). In Figure 1 we illustrate the schematics of our transformer based Multi-Decoder ST model which can also be summarized as follows: h EASR = ENCODERASR ( x ) (7) h DASR l = DECODERASR ( h EASR ,  y ASR 1: l (cid:57) 1 ) (8) h EST = ENCODERST ( h DASR ) (9) h DST l = DECODERST ( h EST ,  y ST 1: l (cid:57) 1 ) (10) As we can see from Equations 9 and 10, the MT sub-network attends only to the decoder representations, h DASR , of the ASR sub-network, which could lead to the error propagation issues from the ASR sub-network to the MT sub-network similar to the cascade systems, as mentioned in 1. To alleviate this problem, we modify equation 10 such that DECODERST attends to both h EST and h EASR : h DSAST l = DECODERSAST ( h EST , h EASR ,  y ST 1: l (cid:57) 1 ) (11) We use the multi-sequence cross-attention discussed by Helcl et al. (2018), shown on the right side of Figure 1, to condition the final outputs generated by h DST l on both speech and transcript information in an attempt to allow our network to recover from intermediate mistakes during inference. We call this model the Multi-Decoder w/ Speech-Attention. For our baseline model, we use an end-to-end encoder-decoder (Enc-Dec) ST model with ASR joint training (Inaguma et al., 2020) as an auxiliarly loss to the speech encoder. In other words, the model consumes speech input using the ENCODERASR , to produce h EASR , which is used for cross-attention by DECODERASR and the DECODERST . Using the decomposed ASR task as an auxiliary loss also helps the baseline Enc-Dec model and provide strong baseline performance, as we will see in Section 6. Data: We demonstrate the efficacy of our proposed approach on ST in the Fisher-CallHome corpus corpus (Post et al., 2013) which contains 170 hours of Spanish conversational telephone speech, transcriptions, and English translations. All punctuations except apostrophes were removed and results are reported in terms of detokenized case-insensitive BLEU (Papineni et al., 2002; Post, 2018). We compute BLEU using the 4 references in Fisher (dev, dev2, and test) and the single reference in CallHome (dev and test) (Post et al., 2013; Kumar et al., 2014; Weiss et al., 2017). We use a joint source and target vocabulary of 1K byte pair encoding (BPE) units (Kudo and Richardson, 2018). We prepare the corpus using the ESPnet library and we follow the standard data preparation, where inputs are globally mean-variance normalized log-mel filterbank and pitch features from up-sampled 16kHz audio (Watanabe et al., 2018). We also apply speed perturbations of 0.9 and 1.1 and the SS SpecAugment policy (Park et al., 2019). Baseline Configuration: All of our models are implemented using the ESPnet library and trained on 3 NVIDIA Titan 2080Ti GPUs for  12 hours. For the Baseline Enc-Dec baseline, discussed in 4, we use an ENCODERASR consisting of a convolutional sub-sampling by a factor of 4 (Watan-abe et al., 2018) and 12 transformer encoder blocks with 2048 feed-forward dimension, 256 attention dimension, and 4 attention heads. The DECODERASR and DECODERST both consist of 6 transformer decoder blocks with the same configuration as ENCODERASR . There are 37.9M trainable parameters. We apply dropout of 0.1 for all components, detailed in the Appendix (A.1). We train our models using an effective batch-size of 384 utterances and use the Adam optimizer (Kingma and Ba, 2015) with inverse square root decay learning rate schedule. We set learning rate to 12.5, warmup steps to 25K, and epochs to 50. We use joint training with hybrid CTC/attention ASR (Watanabe et al., 2017) by setting mtl-alpha to 0 . 3 and asr-weight to 0 . 5 as defined by Watanabe et al. (2018). During inference, we perform beam search (Seki et al., 2019) on the ST sequences, using a beam size of 10, length penalty of 0.2, max length ratio of 0.3 (Watanabe et al., 2018). Multi-Decoder Configuration: For the Multi-Decoder ST model, discussed in 3, we use the same transformer configuration as the baseline for the ENCODERASR , DECODERASR , and DECODERST . Additionally, the Multi-Decoder has an ENCODERST consisting of 2 transformer encoder blocks with the same configuration as ENCODERASR , giving a total of 40.5M trainable parameters. The training configuration is also the same as for the baseline. For the Multi-Decoder w/ Speech-Attention model (42.1M trainable parame-ters), we increase the attention dropout of the ST decoder to 0.4 and dropout on all other components of the ST decoder to 0.2 while keeping dropout on the remaining components at 0.1. We verified that increasing the dropout does not help the vanilla multi-decoder ST model. During inference, we perform beam search on both the ASR and ST output sequences, as discussed in 3. The ST beam search is identical to that of the baseline. For the intermediate ASR beam search, we use a beam size of 16, length penalty of 0.2, max length ratio of 0.3. In some of our experiments, we also include fusion of a source language LM with a 0.2 weight and CTC with a 0.3 weight to re-score the intermediate ASR beam search (Watanabe et al., 2017). For the Speech-Attention variant, we increase LM weight to 0.4. Note that the ST beam search configuration remains constant across our baseline and Multi-Decoder experiments as our focus is on improving overall performance through searchable intermediate representations. Thus, the various re-scoring techniques applied to the ASR beam search are options newly enabled by our proposed architecture and are not used in the ST beam search. Table 1 presents the overall ST performance (BLEU) of our proposed Multi-Decoder model. Our model improves by +2.9/+0.3 (Fisher/CallHome) over the best cascaded baseline and by +5.6/+1.5 BLEU over the best published end-to-end baselines. With Speech-Attention, our model improves by +3.4/+1.6 BLEU over the cascaded baselines and +7.1/+2.8 BLEU over encoder-decoder baselines. Both the Multi-Decoder and Multi-Decoder w/ Speech-Attention on average are further improved by +0.9/+0.4 BLEU through ASR re-scoring. 4 Table 1 also includes our implementation of the Baseline Enc-Dec model discussed in 4. In this way, we are able to make a fair comparison with our framework as we control the model and inference 4 We also evaluate our models using other MT metrics to supplement these results, as shown in the Appendix (A.2). configurations to be analagous. For instance, we keep the same search parameters for the final output in the baseline and the Multi-Decoder to demonstrate impact of the intermediate beam search. An added benefit of our proposed approach over the Baseline Enc-Dec is the ability to monitor the individual performances of the ASR (% WER) and MT (BLEU) sub-nets as shown in Table 2. The Multi-Decoder w/ Speech-Attention shows a greater MT sub-net performance than the Multi-Decoder as well as a slight improvement of the ASR sub-net, suggesting that ST can potentially help ASR. The overall ST performance improves when a higher beam size is used in the intermediate ASR search, and this increase can be attributed to the improved ASR sub-net performance. Figure 1 shows this trend across ASR beam sizes of 1, 4, 8, 10, 16 while fixing the ST decoding beam size to 10. A 1 4 8 10 16 51 . beam size of 1, which is a greedy search, results in lower ASR sub-net and overall ST performances. As beam sizes become larger, gains taper off as can be seen between beam sizes of 10 and 16. External models like CTC acoustic models and language models are commonly used for re-scoring encoder-decoder models (Hori et al., 2017), due to the difference in their modeling capabilities. CTC directly models transcripts while being conditionally independent on the other outputs given the input, and LMs predict the next token in a sequence. Both variants of the Multi-Decoder improve due to improved ASR sub-net performance using exter-Overall Sub-Net Model ST(  ) ASR(  ) Multi-Decoder 52.7 22.6 +ASR Re-scoring w/ LM 53.2 22.6 +ASR Re-scoring w/ CTC 52.8 22.1 +ASR Re-scoring w/ LM 53.3 21.7 Multi-Decoder w/ Speech-Attn. nal CTC and LM models for re-scoring, as shown in Table 3. We use a recurrent neural network LM trained on the Fisher-CallHome Spanish transcripts with a dev perplexity of 18.8 and the CTC model from joint loss applied during training. Neither external model incorporates additional data. Although the impact of the LM-only re-scoring is not shown in the ASR % WER, it reduces substitution and deletion rates in the ASR and this is observed to help the overall ST performance. As discussed in 3, our Multi-Decoder model inherits the error propagation issue as can be seen in Figure 3. For the easiest bucket of utterances with < 40% WER in Multi-Decoder's ASR subnet, our model's ST performance, as measured by the corpus BLEU of the bucket, exceeds that of the Baseline Enc-Dec. The inverse is true for the more difficult bucket of [40 , 80)% , showing that error propagation is limiting the performance of our model; however, we show that multi-sequence attention can alleviate this issue. For extremely difficult utterances in the  80% bucket, ST performance for all three approaches is suppressed. We also provide qualitative examples of error propagation avoidance in the Appendix (A.3). In this section, we discuss the generalizability of our framework towards out-of-domain data. We also extend our Multi-Decoder model to other sequence tasks like speech recognition. Finally, we apply our ST models to a larger corpus with more language pairs and a different domain of speech. Like cascaded systems, searchable intermediates provide our model adaptability in individual subsystems towards out-of-domain data using external in-domain language model, thereby giving access to more in-domain data. Specifically for speech translation systems, this means we can use in-domain language models in both source and target languages. We test the robustness of our Multi-Decoder model trained on Fisher-CallHome conversational speech dataset on read speech CoVost-2 dataset (Wang et al., 2020b). In Table 4 we show that re-scoring the ASR sub-net with an in-domain LM improves ASR with around 10.0% lower WER, improving the overall ST performance by around +2.5 BLEU. Compared to an in-domain ST baseline (Wang et al., 2020a), our out-of-domain Multi-Decoder with in-domain ASR re-scoring demonstrates the robustness of our approach. We apply our generic framework to another decomposable sequence task, speech recognition, and show the results of various levels of decomposition in Table 5. We show that with phoneme, character, or byte-pair encoding (BPE) sequences as intermediates, the Multi-Decoder presents strong results on both Fisher and CallHome test sets. We also observe that the BPE intermediates perform bet-Overall Sub-Net Model ST(  ) ASR(  ) IN-DOMAINST MODEL Baseline (Wang et al., 2020b) 12.0 -+ASR Pretrain (Wang et al., 2020b)  23.0 16.0 OUT-OF-DOMAINST MODEL Multi-Decoder 11.8 46.8 +ASR Re-scoring w/ in-domain LM 14.4 36.7 Multi-Decoder w/ Speech-Attention 12.6 46.5 +ASR Re-scoring w/ in-domain LM 15.0 36.7 Table 4: Results presenting the overall ST performance (BLEU) and the sub-net ASR (% WER) of our Multi-Decoder models when tested on out-of-domain data. In addition to our results using the 170 hours of the Spanish-English Fisher-CallHome corpus, in Table 6 we show that our decompositional framework is also effective on larger ST corpora. In particular, we use 400 hours of English-German and 500 hours of English-French ST from the MuST-C corpus (Di Gangi et al., 2019). Our Multi-Decoder model improves by +2.7 and +1.5 BLEU, in German and French respectively, over end-to-end baselines from prior works that do not use additional training data. We show that ASR re-scoring gives an additional +0.1 and +0.4 BLEU improvement. 5 By extending our Multi-Decoder models to this MuST-C study, we show the generalizability of our 5 Details of the MuST-C data preparation and model parameters are detailed in Appendix (A.4). approach across several dimensions of ST tasks. First, our approach consistently improves over baselines across multiple language-pairs. Second, our approach is robust to the distinct domains of telephone conversations from Fisher-CallHome and the TED-Talks from MuST-C. Finally, by scaling from 170 hours of Fisher-CallHome data to 500 hours of MuST-C data, we show that the benefits of decomposing sequence tasks with searchable hidden intermediates persist even with more data. Furthermore, the performance of our Multi-Decoder models trained with only English-German or English-French ST data from MuST-C is comparable to other methods which incorporate larger external ASR and MT data in various ways. For instance, Zheng et al. (2021) use 4700 hours of ASR data and 2M sentences of MT data for pretraining and multi-task learning. Similarly, Bahar et al. (2021) use 2300 hours of ASR data and 27M sentences of MT data for pretraining. Our competitive performance without the use of any additional data highlights the data-efficient nature of our proposed end-to-end framework as opposed to the baseline encoder-decoder model, as pointed out by Sperber and Paulik (2020). Compositionality: A number of recent works have constructed composable neural network modules for tasks such as visual question answering (Andreas et al., 2016), neural MT (Raunak et al., 2019), and synthetic sequence-to-sequence tasks (Lake, 2019). Modules that are first trained separately can subsequently be tightly integrated into a single end-to-end trainable model by passing differentiable soft decisions instead of discrete decisions in the intermediate stage (Bahar et al., 2021). Further, even a single encoder-decoder model can be decomposed into modular components where the encoder and decoder modules have explicit functions (Dalmia et al., 2019). Joint Training with Sub-Tasks: End-to-end sequence models been shown to benefit from introducing joint training with sub-tasks as auxiliary loss functions for a variety of tasks like ASR (Kim et al., 2017), ST (Salesky et al., 2019; Liu et al., 2020a; Dong et al., 2020; Le et al., 2020), SLU (Haghani et al., 2018). They have been shown to induce structure (Belinkov et al., 2020) and improve the model performance (Toshniwal et al., 2017), but this joint training may reduce data efficiency if some sub-nets are not included in the final end-to-end model (Sperber et al., 2019; Wang et al., 2020c). Our framework avoids this sub-net waste at the cost of computational load during inference. Speech Translation Decoders: Prior works have used ASR/MT decoding to improve the overall ST decoding through synchronous decoding (Liu et al., 2020a), dual decoding (Le et al., 2020), and successive decoding (Dong et al., 2020). These works partially or fully decode ASR transcripts and use discrete intermediates to assist MT decoding. Tu et al. (2017) and Anastasopoulos and Chiang (2018) are closest to our multi-decoder ST model, however the benefits of our proposed framework are not entirely explored in these works. Two-Pass Decoding: Two-pass decoding involves first predicting with one decoder and then re-evaluating with another decoder (Geng et al., 2018; Sainath et al., 2019; Hu et al., 2020; Rijh-wani et al., 2020). The two decoders iterate on the same sequence, so there is no decomposition into sub-tasks in this method. On the other hand, our approach provides the subsequent decoder with a more structured representation than the input by decomposing the complexity of the overall task. Like two-pass decoding, our approach provides a sense of the future to the second decoder which allows it to correct mistakes from the previous first decoder. Auto-Regressive Decoding: As auto-regressive decoders inherently learn a language model along with the task at hand, they tend to be domain spe-cific (Samarakoon et al., 2018; Mller et al., 2020). This can cause generalizability issues during inference (Murray and Chiang, 2018; Yang et al., 2018), impacting the performance of both the task at hand and any downstream tasks. Our approach alleviates these problems through intermediate search, external models for intermediate re-scoring, and multi-sequence attention. We present searchable hidden intermediates for end-to-end models of decomposable sequence tasks. We show the efficacy of our Multi-Decoder model on the Fisher-CallHome Es  En and MuST-C En  De and En  Fr speech translation corpora, achieving state-of-the-art results. We present various benefits in our framework, including sub-net performance monitoring, beam search for better hidden intermediates, external models for better search, and error propagation avoidance. Further, we demonstrate the flexibility of our framework towards out-of-domain tasks with the ability to adapt our sequence model at intermediate stages of decomposition. Finally, we show generalizability by training Multi-Decoder models for the speech recognition task at various levels of decomposition. We hope insights derived from our study stimulate research on tighter integrations between the benefits of cascaded and end-to-end sequence models. Exploiting searchable intermediates through beam search is just the tip of the iceberg for search algorithms, as numerous approximate search techniques like diverse beam search (Vijayakumar et al., 2018) and best-first beam search (Meister et al., 2020) have been recently proposed to improve diversity and approximation of the most-likely sequence. Incorporating differentiable lattice based search (Hannun et al., 2020) can also allow the subsequent sub-net to digest n-best representations. This work started while Vikas Raunak was a student at CMU, he is now working as a Research Scientist at Microsoft. We thank Pengcheng Guo, Hirofumi Inaguma, Elizabeth Salesky, Maria Ryskina, Marta Mndez Simn and Vijay Viswanathan for their helpful discussion during the course of this project. We also thank the anonymous reviewers for their valuable feedback. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) (Towns et al., 2014), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system (Nystrom et al., 2015), which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). The work was supported in part by an AWS Machine Learning Research Award. This research was also supported in part the DARPA KAIROS program from the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation there on. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain objective method method abstain objective objective result result other other other abstain other other other other abstain abstain other other other other other method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain other other abstain method abstain abstain method abstain abstain method method method method method method abstain abstain method method method method abstain abstain method method abstain abstain method method method abstain method method method method method abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result result result abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result method objective result abstain abstain abstain other other other other other other other other other