Over the last few years, there has been a move towards data curation for multilingual task-oriented dialogue (ToD) systems that can serve people speaking different languages. However, existing multilingual ToD datasets either have a limited coverage of languages due to the high cost of data curation, or ignore the fact that dialogue entities barely exist in countries speaking these languages. To tackle these limitations, we introduce a novel data curation method that generates GlobalWoZ  a large-scale multilingual ToD dataset globalized from an English ToD dataset for three unexplored use cases of multilingual ToD systems. Our method is based on translating dialogue templates and filling them with local entities in the target-language countries. Besides, we extend the coverage of target languages to 20 languages. We will release our dataset and a set of strong baselines to encourage research on multilingual ToD systems for real use cases. 1 1 Introduction One of the fundamental objectives in pursuit of artificial intelligence is to enable machines with the ability to intelligently communicate with human in natural languages, with one of the widely-heralded applications being the task-oriented dialogue (ToD) systems (Gupta et al., 2006; Bohus and Rudnicky, 2009). Recently, ToD systems have been successfully deployed to assist users with accomplishing certain domain-specific tasks such as hotel booking, alarm setting or weather query (Eric et al., 2017; Wu et al., 2019; Lin et al., 2020; Zhang et al., 2020), thanks to the joint advent of neural networks and availability of domain-specific data. However, most existing ToD systems are predominately built for English, limiting their service for all of the  Bosheng Ding is under the Joint PhD Program between Alibaba and Nanyang Technological University. world's citizens. The reason of this limitation lies in the stark lack of high-quality multilingual ToD datasets due to the high expense and challenges of human annotation (Razumovskaia et al., 2021). One solution to this is annotating conversations in other languages from scratch, e.g., CrossWoZ (Zhu et al., 2020) and BiToD (Lin et al., 2021). However, these methods involve expensive human efforts for dialogue collection in the other languages, resulting in a limited language/domain coverage. The other major line of work focused on translating an existing English ToD dataset into target languages by professional human translators (Upadhyay et al., 2018; Schuster et al., 2019; van der Goot et al., 2021; Li et al., 2021). Despite the increasing language coverage, these methods simply translated English named entities (e.g., location, restaurant name) into the target languages, while ignored the fact that these entities barely exist in countries speaking these languages. This hinders a trained ToD system from supporting the real use cases where a user looks for local entities in a target-language country. For example in Figure 1, a user may look for the British Museum when traveling to London (A.), while look for the Oriental Pearl Tower when traveling to Shanghai (B.). In addition, prior studies (Cheng and Butler, 1989; Kim, 2006) have shown that code-switching phenomena frequently occurs in a dialogue when a speaker cannot express an entity immediately and has to alternate between two languages to convey information more accurately. Such phenomena could be ubiquitous during the cross-lingual and cross-country task-oriented conversations. One of the reasons for code-switching is that there are no exact translations for many local entities in the other languages. Even though we have the translations, they are rarely used by local people. For example in Figure 1 (C.), after obtaining the recommendation from a ToD system, a Chinese speaker traveling to London would rather use the English 1639 London A. Use Case: E&E I'm looking for an attraction to visit in London. entity British Museum than its Chinese translation to search online or ask local people. To verify this code-switching phenomena, we have also conducted a case study (6.1) which shows that searching the information about translated entities online yields a much higher failure rate than searching them in their original languages. Motivated by these observations, we define three unexplored use cases 2 of multilingual ToD where a foreign-language speaker uses ToD in the foreign-language country ( F&F ) or an English country ( F&E ), and an English speaker uses ToD in a foreign-language country ( E&F ). These use cases are different from the traditional E&E use case where an English speaker uses ToD in an English-speaking country. To bridge the aforementioned gap between existing data curation methods and the real use cases, we propose a novel data curation method that global-izes an existing multi-domain ToD dataset beyond English for the three unexplored use cases. Specifically, building on top of MultiWoZ (Budzianowski et al., 2018)  an English ToD dataset for dialogue state tracking (DST), we create GlobalWoZ, a new multilingual ToD dataset in three new target-languages via machine translation and crawled ontologies in the target-language countries. Our method only requires minor human efforts to post-edit a few hundred machine-translated dialogue templates in the target languages for evaluation. Besides, as cross-lingual transfer via pre-2 See comparisons of these use cases in Appendix A trained multilingual models (Devlin et al., 2019; Conneau et al., 2020; Liu et al., 2020; Xue et al., 2021) has proven effective in many cross-lingual tasks, we further investigate another question: How do these multilingual models trained on the English ToD dataset transfer knowledge to our globalized dataset? To answer this question, we prepare a few baselines by evaluating popular ToD systems on our created test datasets in a zero-shot cross-lingual transfer setting as well as a few-shot setting. Our contributions include the following:  To the best of our knowledge, we provide the first step towards analyzing three unexplored use cases for multilingual ToD systems. We propose a cost-effective method that creates a new multilingual ToD dataset from an existing English dataset. Our dataset consists of high-quality test sets which are first translated by machines and then post-edited by professional translators in three target languages (Chinese, Spanish and Indonesian). We also leverage machine translation to extend the language coverage of test data to another 17 target languages. Our experiments show that current multilingual systems and translate-train methods fail in zero-shot cross-lingual transfer on the dialogue state tracking task. To tackle this problem, we propose several data augmentation methods to train strong baseline models in both zero-shot and few-shot cross-lingual transfer settings. In order to globalize an existing English ToD dataset for the three aforementioned use cases, we propose an approach consisting of four steps as shown in Figure 2: (1) we first extract dialogue templates from the English ToD dataset by replacing English-specific entities with a set of general-purpose placeholders (2.1); (2) we then translate the templates to a target language for both training and test data, with one key distinction that we only post-edit the test data by professional translators to ensure the data quality for evaluation (2.2); (3) next, we collect ontologies (Kiefer et al., 2021) containing the definitions of dialogue acts, local entities and their attributes in the target-language countries (2.3); (4) finally, we tailor the translated templates by automatically substituting the placeholders with entities in the extracted ontologies to construct data for the three use cases (2.4). 2.1 Automatic Template Creation We start with MultiWoZ 2.2 (Zang et al., 2020)  a high-quality multi-domain English ToD dataset with more accurate human annotations compared to its predecessors MultiWoZ 2.0 (Budzianowski et al., 2018) and MultiWoz 2.1 (Eric et al., 2020). For the sake of reducing human efforts for collecting ToD context in the target languages, we re-use the ToD context written by human in MultiWoZ as the dialogue templates. Specifically as shown in Figure 2, we replace the English entities in MultiWoz by a set of general-purpose placeholders such as [attraction-name0] and [attraction-postcode1] , where each placeholder contains the entity's domain, attribute and ID. To do so, we first build a dictionary with entity-placeholder pairs by parsing the annotations of all dialogues. For example, from a dialogue text  I recommend Whale of a time and the post code is cb238el. , we obtain two entity-placeholder pairs from its human annotations, i.e., ( Whale of a time , [attraction-name0] ) and ( cb238el , [attraction-postcode1] ). Next, we identify entities in the dialogue by their word index from the human annotations, replace them with their placeholders in the dictionary, and finally obtain dialogue templates with placeholders. Notably, we skip the entities with their attributes of [choice] and [ref] that represent the number of choices and booking reference number, as these attributes could be used globally. Following Liu et al. (2021) that translates sentences with placeholders, we use a machine translation system 3 to translate dialogue templates with our designed placeholders. As we observe, a placeholder containing an entity domain, attribute and ID (e.g., attraction-name0 ) is useful to provide contextually meaningful information to the translation system, thus usually resulting in a high-quality translation with the placeholder unchanged 4 . This also enables us to easily locate the placeholders in the translation output and replace them with new entities in the target language. To build a high-quality test set for evaluation, we further hire professional translators to post-edit a few hundred machine-translated templates, which produces natural and coherent sentences in the target languages. 5 With the goal of selecting representative test templates for post-editing, we first calculate the frequency of all the 4-gram combinations in the MultiWoZ data, and then score each dialogue in the test set by the sum of the frequency of all the 4-gram combinations in the dialogue divided by the dialogue's word length. We use this scoring function to estimate the representiveness of a dialogue in the original dataset. Finally, we select the top 500 high-scoring dialogues in the test set for post-editing. 6 We also use the same procedure to create a small high-quality training set for few-shot cross-lingual transfer setting. Meanwhile, we crawl the attribute information of local entities in three cities from public websites (e.g., tripadvisor.com, booking.com) to create three ontologies for the three corresponding target languages respectively. As shown in Table 8 in Appendix E, we select Barcelona for Spanish (an Indo-European language), Shanghai for Mandarin (a Sino-Tibetan language) and Jakarta for Indonesian (an Austronesian language), which cover a set of typologically different language families. Given a translated dialogue template, we can easily sample a random set of entities for a domain of interest from a crawled ontology and assign the entities to the template's placeholders to obtain a 3 We use Google Translate ( https://cloud.google. com/translate ), an off-the-shelf MT system. 4 Appendix B has an example of label sequence translation. 5 Appendix C shows the bleu scores between MT test data and MTPE test data. 6 Appendix D shows the English test data distribution. Repeating this procedure on each dialogue template, we can easily build a high-quality labeled dataset in the target language. Table 9 in Appendix F shows the statistics of our collected entities in the target languages compared with the English data. The number of our collected entities are either larger than or equal to those in the English data except for the train domain; we collected the information about only 100 trains for each languages due to the complexity in collecting relevant information. After the above steps, we assign entities in a target language to the translated templates in the same target language for the F&F case, while assigning target-language entities to the English (source-language) templates for the F&E case. As for the E&F case, we keep the original English context by skipping the translation step and replace the placeholders with local entities in the target language (see Figure 2 for examples). To sum up, our proposed method has three key properties: (1) our method is cost-effective as we only require a limited amount of post-editing efforts for a test set when compared to the expensive crowd-sourced efforts from the other studies; (2) we can easily sample entities from an ontology to create large-scale machine-translated data as a way of data augmentation for training; (3) our method is flexible to update entities in a ToD system whenever an update of ontology is available, e.g., extension of new entities. We refer the readers to Table 10 for the data statistics of GlobalWoZ and Figure 9 for dialogue examples in the appendix. Our experiments focus on the dialogue state tracking (DST), one of the fundamental components in a ToD system that predicts the goals of a user query in multi-turn conversations. We follow the setup in MultiWoZ (Budzianowski et al., 2018) to evaluate ToD systems for DST by the joint goal accuracy which measures the percentage of correctly predicting all goals in a multi-turn conversation. Zero-Shot Cross-lingual Transfer: Unlike prior studies that annotate a full set of high-quality training data for a target language, we investigate the zero-shot cross-lingual transfer setting where we have access to only a high-quality human-annotated English ToD data (referred to as gold standard data hereafter). In addition, we assume that we have access to a machine translation system that translates from English to the target language. We investigate this setting to evaluate how a multilingual ToD system transfers knowledge from a high-resource source language to a low-resource target language. Few-Shot Cross-lingual Transfer: We also investigate few-shot cross-lingual transfer, a more practical setting where we are given a small budget to annotate ToD data for training. Specifically, we include a small set (100 dialogues) of high-quality training data post-edited by professional translators 1642 (2.2) in a target language, and evaluate the efficiency of a multilingual ToD on learning from a few target-language training examples. We prepare a base model for GlobalWoZ in the zero-shot and few-shot cross-lingual transfer settings. We select Transformer-DST (Zeng and Nie, 2020) as our base model as it is one of the state-of-the-art models on both MultiWoZ 2.0 and MultiWoZ 2.1 7 . In our paper, we replace its BERT encoder with an mBERT encoder (Devlin et al., 2019) for our base model and propose a series of training methods for GlobalWoZ. As detailed below, we propose several data augmentation baselines that create different training and validation data for training a base model. Note that all the proposed baselines are model agnostic and the base model can be easily substituted with other popular models (Heck et al., 2020; Lin et al., 2020). For each baseline, we first train a base model on its training data for 20 epochs and use its validation set to select the best model during training. Finally we evaluate the best model of each baseline on the same test set from GlobalWoZ. We will release GlobalWoZ and our pre-trained models to encourage faster adaptation to future research. We refer the readers to Table 11 and Table 12 in Appendix I while reading the subsequent methods for a better understanding. We train a base model on the gold standard English data (E&E) and directly apply the learned model to the test data of the three use cases in GlobalWoZ. With this method, we simulate the condition of having labeled data only in the source language for training, and evaluate how the model transfers knowledge from English to the three use cases. We use Zero-Shot (E&E) to denote this method. We use our data curation method (2) to translate the templates by an MT system but replace the placeholders in the translated templates with machine-translated entities to create a set of pseudo-labeled training data. Next, we train a base model on the translated training data without local entities, and evaluate the model on the three use cases. We denote this method as Translate-Train . 7 According to the leaderboards of Multi-domain Dialogue State Tracking on MultiWoZ 2.0 and MultiWoZ 2.1 on paperwithcode.com as of 11/15/2021. By skipping the human post-editing step in our data curation method (2), we leverage a machine translation system to automatically create a large set of pseudo-labeled training data with local entities for the three use cases. In the F&F case, we translate the English templates by the MT system and replace the placeholders in the translated templates with foreign-language entities to create a training dataset. In the F&E case, we replace the placeholders in the translated templates with the original English entities to create a code-switched training dataset. In the E&F case, we use the original English templates and replace the placeholders in the English templates with foreign-language entities to create a code-switch training dataset. With this data augmentation method, we can train a base model on each pseudo-labeled training dataset created for each use case. We denote this method as SUC (Single-Use-Case). We investigate the performance of combining the existing English data and the pseudo-labeled training data created for one of the three use cases (i.e., F&F, F&E, E&F), one at a time, to do bi-use-case training. In the bilingual training, we only combine the gold English data (E&E) with the pseudo-labeled training data in one target language in one use case for joint training. We denote this method as BBUC (Bilingual Bi-Use-Case). In the multilingual training, we combine gold English data (E&E) and pseudo-labeled training data in all languages in one use case for joint training. We denote this method as MBUC (Multilingual Bi-Use-Case). We also propose to combine the existing English data (E&E) and all the pseudo-labeled training data in all target languages for all the use cases (F&F, F&E, E&F). We then train a single model on this combined multilingual training dataset and evaluate the model on test data in all target languages for all three use cases . We denote this method as MMUC (Multilingual Multi-Use-Case). In this section, we show the results of all methods in the zero-shot (5.1) and few-shot (5.2) settings. Table 1 reports the joint goal accuracy of all proposed methods on the three different sets of test data in the F&F, F&E, and E&F use cases 8 . Both Zero-Shot (E&E) and Translate-Train struggle, achieving average accuracy of less than 10 in all use cases. Despite its poor performance, Zero-Shot (E&E) works much better in F&E than F&F, while its results in F&F and E&F are comparable, indicating that a zero-shot model trained in E&E can transfer knowledge about local English entities more effectively than knowledge about English context in downstream use cases. Besides, we also find that Zero-Shot (E&E) performs better on the Spanish or Indonesian context than the Chinese context in F&E. One possible reason is that English is closer to the other Latin-script languages (Spanish and Indonesian) than Chinese. Our proposed data augmentation methods (SUC, BBUC, MBUC) perform much better than non-adapted methods (Zero-Shot (E&E) and Translate-Train) that do not leverage any local entities for training. In particular, it is worth noting that even though Translate-Train and SUC both do training on foreign-language entities in F&F and E&F, there is a huge gap between these two methods, since Translate-Train has only access to the machine-translated entities rather than the real local entities used by SUC. This huge performance gaps not only show that Translate-Train is not an effective method in practical use cases but also prove that having access to local entities is a key to building a multilingual ToD system for practical usage. Comparing our data augmentation methods SUC and BBUC, we find that the base model can benefit from training on additional English data (E&E), especially yielding a clear improvement of up to 5.58 average accuracy points in F&E. Moreover, when we increase the number of languages in the bi-use-case data augmentations (i.e., MBUC), we observe an improvement of around 1 average accuracy points in all three use cases w.r.t. BBUC. These observations encourage a potential future direction that explores better data augmentation methods to create high-quality pseudo-training data. Notice that we can train a single model by MMUC for all use cases rather than training separate mod-8 els, one for each use case. In Figure 3, we compare MMUC and MBUC (rows) on the test data in the four use cases (columns). Although MMUC may not achieve the best results in each use case, it achieves the best average result over the four use cases, indicating the potential of using one model to simultaneously handle all the four use cases. In few-shot experiments, we use the same scoring function based on frequency of all 4-gram combinations (2.2) to select 100 additional dialogues from train set for human-post editing, and create high-quality training data for each of the three use cases. To avoid overfitting on this small few-shot dataset, we combine the few-shot data with the existing English data for training a base model (Few-Shot+Zero-Shot (E&E)). Next, we also investigate a model trained with additional synthetic data created by our proposed SUC. In Figure 4, we find that our proposed SUC without additional few-shot 1644 F&F F&E E&F 0 10 20 30 40 50 60 1.28 9.12 1.77 11.72 40.84 17.78 28.96 48.71 36.79 34.16 55.15 36.88 34.70 56.77 37.61 Zero Shot (E&E) Few Shot+Zero Shot (E&E) SUCFew Shot+SUC Few Shot+Zero Shot (E&E)+SUC Figure 4: Few-shot cross-lingual average joint accuracy on DST over three target languages in three use cases. data has already outperformed the model trained with few-shot data and English data (Few-shot + Zero-Shot (E&E)), indicating that the model benefit more from a large amount of pseudo-labeled data than a small set of human-labeled data. If we combine the data created by SUC with the few-shot data or with both few-shot and English data to train the model, we observe improvements over SUC, especially with a clear gain of 8.06 accuracy points in F&E. We refer the readers to Table 14 in the appendix for detailed scores in all target languages. One key research question is to validate whether code-switched use cases with local entities (i.e., F&E, E&F) are practically more useful for information seeking. To answer this question, we compare the failure rate of using local entities and machine-translated entities in information search, which is a proxy to the efficiency of using these two types of entities in conversations. We first randomly select 100 entities (33 attractions, 33 hotels and 34 restaurants) of Cambridge, Shanghai, Barcelona and Jakarta. We translate the English entities into Mandarin, Spanish and Indonesian and the foreign-language entities into English via Google Translate. We then manually search the translated entities on Google to check whether we can find the right information of the original entities. Notice that the failure of the above verification partially come from the translation error made by Google Translate, or the search failure due to the fact that this entity does not have a bilingual version at all. In Table 2, we observe a high failure rate of around 60% for almost all translated directions (except Zh  En) Translate Search En  Zh En  Es En  Id Zh  En Es  En Id  En (cid:34) (cid:34) 35 42 36 62 30 31 (cid:34) (cid:37) 61 34 51 18 18 15 (cid:37) (cid:34) 0 24 13 11 50 54 (cid:37) (cid:37) 4 0 0 8 2 0 Failure Case (MTed Entities) 65 58 64 37 70 69 Failure Rate (MTed Entities) 65% 58% 64% 37% 70% 69% Failure Rate (Original Entities) 3% 3% 3% 0% 1% 0% Table 2: The search and translation results of 100 translated entities on Google. due to translation and search failures, significantly exceeding the low failure rate of searching original entities online. Besides, even if we can find the right information of the translated entities, local people may not recognize or use the translated entities for communication, thus this results in in-efficient communication with local people. In previous translation-based work, a multilingual ToD system is usually built based on the translation of English training data (Translate-Train), and is evaluated on translated test data without any local entities (Translate-Test). To verify whether this procedure is reliable to build a multilingual ToD system, we also create a test dataset with translated entities instead of local entities in the target languages. As shown in Figure 5, we find the Translate-Train model performs well on the test data with translated entities, but performs badly on the test data with real local entities. To the best of our knowledge, we provide the first analysis to identify this performance gap between the translated test data and data with real local entities in a more realistic use case 9 . Our work sheds light on the development of a globalized multilingual ToD system in practical use cases. We can tackle 9 Please refer to Appendix L for concrete examples where Translate-Train fails in predicting real local entities. the challenge of localization issues by exploring new data augmentation method. Alternatively we can also explore new methods from the model level by building modular network to update the entities or perform transfer learning to adapt to new case without retraining. We compare the impact of training a model on data with either local contexts or local entities when the model is evaluated on monolingual test data in F&F and E&E. Specifically, when the train set has access to local context only, all the entities in the train set are replaced by entities in non-target languages. Similarly, when the train set has access to local entities only, the contexts in the train set are replaced by context in the non-target languages. Table 3 shows that both local contexts and local entities are essential to building ToD systems in the target language. A further analysis in Table 15 and Table 16 in the appendix shows that training with local entities is more important if the entities and contexts are written in the same type of language script (e.g. Latin script). With our proposed data curation method, it is possible to extend the dataset to cover more languages without spending extra costs if we skip the human post-editing step. Before doing so, one key question is whether the evaluation on the translated data without human post-editing is reliable as a proxy of the model performance. Thus, we conduct the experiments by evaluating the model performance of all baselines (4) on two sets of test data built with local entities: (1) MT test data where translated template is created by machine translation only (2.2); (2) MTPE test data where translated template is first translated by machines and post-edited later by professional translators. As shown in Table 4, the overall reported results on MT test data are higher than those reported on MTPE test data, which is expected because the distribution of the MT test data is more similar to the MT training Use Case F2F F2E Methods MT Test MTPE Test MT Test MTPE Test Zero-Shot (E&E) 1.29 1.28 9.64 9.12 Translate-Train 3.71 3.65 4.17 3.97 SUC 35.78 28.96 56.15 48.71 BBUC 36.31 29.74 57.84 54.29 MBUC 37.89 30.76 58.76 56.28 Spearman's correlation 1.0 1.0 Table 4: Comparison of average joint accuracy on DST reported on MT test data and MTPE test data for use case F&F and F&E Case Method Avg F&F Zero-Shot (E&E) 1.48 SUC 16.12 F&E Zero-Shot (E&E) 9.03 SUC 34.20 E&F Zero-Shot (E&E) 1.97 SUC 23.40 Table 5: Average results of Zero-Shot (E&E) on test data of F&F, F&E and E&F in 20 languages. data. Although there are some differences on individual languages, the conclusions derived from the evaluations on the MT test data remain the same as those derived from the evaluation on the MTPE test data. We also calculate the Spearman rank correlation coefficient between the average results reported on MTPE test data and MT test data in Table 4, which shows a statistically high correlation between the system performance on the MT test data and MTPE test data 10 . Therefore, we show that the MT test data can be used as a proxy to estimate the model performance on the real test data for more languages. Thus we build MT test data for another 17 languages that are supported by Google Translate, Trip Advisor and Booking.com at the same time, as stated in Table 8 and Table 9 in the appendix. Table 5 shows the results of Zero-Shot (E&E) and SUC on the test data of F&F, F&E and E&F in 20 languages. The results show that the model has the best performance in the F&E use case compared with the other two use cases, which is consistent with our findings in Table 1. 7 Related Work Over the last few years, the success of ToD systems is largely driven by the joint advent of neural network models (Eric et al., 2017; Wu et al., 2019; Lin et al., 2020) and collections of large-10 Table 17 in the appendix shows detailed scores. scale annotation corpora. These corpora cover a wide range of topics from a single domain (e.g., ATIS (Hemphill et al., 1990), DSTC 2 (Henderson et al., 2014), Frames (El Asri et al., 2017), KVRET (Eric et al., 2017), WoZ 2.0 (Wen et al., 2017), M2M (Schatzmann et al., 2007)) to multiple domains (e.g., MultiWoZ (Budzianowski et al., 2018), SGD (Rastogi et al., 2020)). Most notably among these collections, MultiWoZ is a large-scale multi-domain dataset that focuses on transitions between different domains or scenarios in real conversations (Budzianowski et al., 2018). Due to the high cost of collecting task-oriented dialogues, only a few monolingual or bilingual non-English ToD datasets are available (Zhu et al., 2020; Quan et al., 2020; Lin et al., 2021). While there is an increasing interest in data curation for multilingual ToD systems, a vast majority of existing multilingual ToD datasets do not consider the real use cases when using a ToD system to search for local entities in a country. We fill this gap in this paper to provide the first analysis on three previously unexplored use cases. In this paper, we provide an analysis on three unexplored use cases for multilingual task-oriented dialogue systems. We propose a new data curation method that leverages a machine translation system and local entities in target languages to create a new multilingual TOD dataset, GlobalWoZ. We propose a series of strong baseline methods and conduct extensive experiments on GlobalWoZ to encourage research for multilingual ToD systems. Besides, we extend the coverage of languages on multilingual ToD to 20 languages, marking the one step further towards building a globalized multilingual ToD system for all of the world's citizen. In this section, we would like to address the ethical concerns. All the professional translators in this project have been properly compensated. For Chinese and Spanish, we have followed the standard procurement requirements and engaged three translation companies for quality and price comparison. A small sample of the data had been given to them for MTPE and we then compared their translation results. Following that, we selected the company that produced the best sample translation, and submitted the full translation orders according to the agreed price quotations. For Indonesian, three translation companies were also requested to provide sample MTPE, but our quality check found the quality of these samples to be unsatisfactory. So, no company was engaged, and our in-house Indonesian linguistic resources were used instead. These Indonesian linguists were assigned to work on this project during normal working hours and given proper compensation complying with the local labor laws. This research is partly supported by the Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University. All the costs for machine translation post-editing are funded by DAMO Academy, Alibaba Group. We would like to thank the help from our Alibaba colleagues, Haiyun Peng, Zifan Xu and Ruidan He, and our NTU-NLP team member, Chengwei Qin in this work as well.
abstain abstain objective abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result objective abstain objective abstain abstain other method objective objective objective objective result objective other method method abstain abstain abstain abstain method method abstain abstain method abstain method abstain method method abstain other abstain abstain abstain method abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method method method abstain abstain method method method method method abstain abstain method abstain method abstain abstain method abstain result abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain method abstain abstain result abstain result abstain result abstain result result abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain result result method abstain result abstain abstain other abstain abstain abstain abstain method objective objective objective abstain abstain method abstain method abstain method abstain other other other