Generating code-switched text is a problem of growing interest, especially given the scarcity of corpora containing large volumes of real code-switched text. In this work, we adapt a state-of-the-art neural machine translation model to generate Hindi-English code-switched sentences starting from monolingual Hindi sentences. We outline a carefully designed curriculum of pretraining steps, including the use of synthetic code-switched text, that enable the model to generate high-quality code-switched text. Using text generated from our model as data augmentation, we show significant reductions in perplexity on a language modeling task, compared to using text from other generative models of CS text. We also show improvements using our text for a downstream code-switched natural language inference task. Our generated text is further subjected to a rigorous evaluation using a human evaluation study and a range of objective metrics, where we show performance comparable (and sometimes even superior) to code-switched text obtained via crowd workers who are native Hindi speakers. Code-switching (CS) refers to the linguistic phenomenon of using more than one language within a single sentence or conversation. CS appears naturally in conversational speech among multilingual speakers. The main challenge with building models for conversational CS text is that we do not have access to large amounts of CS text that is conversational in style. One might consider using social media text that contains CS and is more readily available. However, the latter is quite different from conversational CS text in its vocabulary (e.g., due to the frequent use of abbreviated slang terms,  Work done while first two authors were students at IIT Bombay. hashtags and mentions), in its sentence structure (e.g., due to character limits in tweets) and in its word forms (e.g., due to transliteration being commonly employed in social media posts). This motivates the need for a generative model of realistic CS text that can be sampled to subsequently train models for CS text. In this work, we tackle the problem of generating high-quality CS text using only limited amounts of real CS text during training. We also assume access to large amounts of monolingual text in the component languages and parallel text in both languages, which is a reasonable assump-tion to make for many of the world's languages. We focus on Hindi-English CS text where the matrix (dominant) language is Hindi and the embedded language is English. 1 Rather than train a generative model, we treat this problem as a translation task where the source and target languages are monolingual Hindi text and Hindi-English CS text, respectively. We also use the monolingual Hindi text to construct synthetic CS sentences using simple techniques. We show that synthetic CS text, albeit being naive in its construction, plays an important role in improving our model's ability to capture CS patterns. We draw inspiration from the large body of recent work on unsupervised machine translation (Lample et al., 2018a,b) to design our model, which will henceforth be referred to as T ranslation for C odeS witching, or TCS. TCS, once trained, will convert a monolingual Hindi sentence into a Hindi-English CS sentence. TCS makes effective use of parallel text when it is available and uses backtranslation-based objective functions with monolingual text. 1 Given the non-trivial effort involved in collecting annotations from professional annotators and crowd workers, we focused on a single language pair (Hindi-English) and leave explorations on more language pairs for future work. Below, we summarize our main contributions: 1. We propose a state-of-the-art translation model that generates Hindi-English CS text starting from monolingual Hindi text. This model requires very small amounts of real CS text, uses both supervised and unsupervised training objectives and considerably benefits from a carefully designed training curriculum, that includes pretraining with synthetically constructed CS sentences. 2. We introduce a new Hindi-English CS text corpus in this work. 2 Each CS sentence is accompanied by its monolingual Hindi translation. We also designed a crowdsourcing task to collect CS variants of monolingual Hindi sentences. The crowdsourced CS sentences were manually verified and form a part of our new dataset. 3. We use sentences generated from our model to train language models for Hindi-English CS text and show significant improvements in perplexity compared to other approaches. 4. We present a rigorous evaluation of the quality of our generated text using multiple objective metrics and a human evaluation study, and they clearly show that the sentences generated by our model are superior in quality and successfully capture naturally occurring CS patterns. Early approaches of language modeling for code-switched text included class-based n -gram models ( Yeh et al.), factored language models that exploited a large number of syntactic and semantic features (Adel et al., 2015), and recurrent neural language models (Adel et al., 2013) for CS text. All these approaches relied on access to real CS text to train the language models. Towards alleviating this dependence on real CS text, there has been prior work on learning code-switched language models from bilingual data (Li and Fung, 2014b,a; Garg et al., 2018b) and a more recent direction that explores the possibility of generating synthetic CS sentences. (Pratapa et al., 2018) presents a technique to generate synthetic CS text that grammatically adheres to a linguistic theory 2 The new dataset and relevant code is available at: https://www.cse.iitb.ac.in/~pjyothi/TCS . of code-switching known as the equivalence constraint (EC) theory (Poplack, 1979; Sankoff, 1998). Lee and Li (2020) proposed a bilingual attention language model for CS text trained solely using a parallel corpus. Another recent line of work has explored neural generative models for CS text. Garg et al. (2018a) use a sequence generative adversarial network (SeqGAN (Yu et al., 2017)) trained on real CS text to generate sentences that are used to aid language model training. Another GAN-based method proposed by Chang et al. (2019) aims to predict the probability of switching at each token. Winata et al. (2018) and Winata et al. (2019) use a sequence-to-sequence model enabled with a copy mechanism (Pointer Network (Vinyals et al., 2015)) to generate CS data by leveraging parallel monolingual translations from a limited source of CS data. Samanta et al. (2019) proposed a hierarchical variational autoencoder-based model tailored for code-switching that takes into account both syntactic information and language switching signals via the use of language tags. (We present a comparison of TCS with both Samanta et al. (2019) and Garg et al. (2018a) in Section 5.2.1.) In a departure from using generative models for CS text, we view this problem as one of sequence transduction where we train a model to convert a monolingual sentence into its CS counterpart. Chang et al. (2019); Gao et al. (2019) use GAN-based models to modify monolingual sentences into CS sentences, while we treat this problem of CS generation as a translation task and draw inspiration from the growing body of recent work on neural unsupervised machine translation models ( Lample et al., 2018a,b) to build an effective model of CS text. The idea of using translation models for code-switching has been explored in early work (Vu et al., 2012; Li and Fung, 2013; Dhar et al., 2018). Concurrent with our work, there have been efforts towards building translation models from English to CS text (Solorio et al., 2021) and CS text to English (Gupta et al., 2021). While these works focus on translating from the embedded language (En-glish) to the CS text or vice-versa, our approach starts with sentences in the matrix language (Hindi) which is the more dominant language in the CS text. Also, ours is the first work, to our knowledge, to repurpose an unsupervised neural machine translation model to translate monolingual sentences 3156 into CS text. Powerful pretrained models like mBART (Liu et al., 2020) have been used for code-mixed translation tasks in concurrent work (Gau-tam et al., 2021). We will further explore the use of synthetic text with such models as part of future work. Figure 1 shows the overall architecture of our model. This is largely motivated by prior work on unsupervised neural machine translation (Lample et al., 2018a,b). The model comprises of three layers of stacked Transformer (Vaswani et al., 2017) encoder and decoder layers, two of which are shared and the remaining layer is private to each language. Monolingual Hindi (i.e. the source language) has its own private encoder and decoder layers (denoted by Enc p 0 and Dec p 0 , respectively) while English and Hindi-English CS text jointly make use of the remaining private encoder and decoder layers (denoted by Enc p 1 and Dec p 1 , respec-tively). In our model, the target language is either English or CS text. Ideally, we would like Enc p 1 and Dec p 1 to be trained only using CS text. However, due to the paucity of CS text, we also use text in the embedded language (i.e. English) to train these layers. Next, we outline the three main training steps of TCS. (I) Denoising autoencoding (DAE). We use monolingual text in each language to estimate language models. In Lample et al. (2018b), this is achieved via denoising autoencoding where an au-toencoder is used to reconstruct a sentence given a noisy version as its input whose structure is altered by dropping and swapping words arbitrarily ( Lample et al., 2018a). The loss incurred in this step is denoted by LDAE and is composed of two terms based on the reconstruction of the source and target language sentences, respectively. (II) Backtranslation (BT): Once the layers are initialized, one can use non-parallel text in both languages to generate a pseudo-parallel corpus of backtranslated pairs (Sennrich et al., 2015). That is, a corpus of parallel text is constructed by translating sentences in the source language via the pipeline, Enc p 0 , Enc sh , Dec sh and Dec p 1 , and translating target sentences back to the source language via Enc p 1 , Enc sh , Dec sh and Dec p 0 . The backtranslation loss LBT is composed of cross-entropy losses from using these pseudo-parallel Enc p 0 <latexit sha1_base64="Qe5FC3z9zYmVnrv25ZeSnbK6vwM=">AAAB+3icbVDLSsNAFL3xWeur1qWbwSK4KkkVdFkUwWUF+4A2hMl00g6dScLMRCwhv+LGhSJu/RF3/o2TmIW2Hhg4nHMvc+7xY86Utu0va2V1bX1js7JV3d7Z3duvHdR7KkokoV0S8UgOfKwoZyHtaqY5HcSSYuFz2vdn17nff6BSsSi81/OYugJPQhYwgrWRvFp9JLCeSpHehCTz0tizM6/WsJt2AbRMnJI0oETHq32OxhFJBA014VipoWPH2k2x1IxwmlVHiaIxJjM8oUNDQyyoctMie4ZOjDJGQSTNCzUq1N8bKRZKzYVvJvOkatHLxf+8YaKDSzdlYZxoam4rPgoSjnSE8iLQmElKNJ8bgolkJisiUywx0aauqinBWTx5mfRaTees2bo7b7SvyjoqcATHcAoOXEAbbqEDXSDwCE/wAq9WZj1bb9b7z+iKVe4cwh9YH99n+JSs</latexit> Enc sh <latexit sha1_base64="2ClzRthuuS4GM27vrwBa0TGgjjA=">AAAB+nicbVDLSsNAFJ3UV62vVJduBovgqiRV0GVRBJcV7APaECbTSTt0ZhJmJkqJ+RQ3LhRx65e482+cxCy09cDA4Zx7mXNPEDOqtON8WZWV1bX1jepmbWt7Z3fPru/3VJRITLo4YpEcBEgRRgXpaqoZGcSSIB4w0g9mV7nfvydS0Ujc6XlMPI4mgoYUI20k366PONJTydNrgTM/VdPMtxtO0ykAl4lbkgYo0fHtz9E4wgknQmOGlBq6Tqy9FElNMSNZbZQoEiM8QxMyNFQgTpSXFtEzeGyUMQwjaZ7QsFB/b6SIKzXngZnMg6pFLxf/84aJDi+8lIo40cScVnwUJgzqCOY9wDGVBGs2NwRhSU1WiKdIIqxNWzVTgrt48jLptZruabN1e9ZoX5Z1VMEhOAInwAXnoA1uQAd0AQYP4Am8gFfr0Xq23qz3n9GKVe4cgD+wPr4BBYiUfg==</latexit> Dec sh <latexit sha1_base64="5yqcHaZeQ2t4ytT0kWqxFfWwFBA=">AAAB+nicbVDLSsNAFL3xWesr1aWbwSK4KkkVdFnUhcsK9gFtCJPppB06k4SZiVJiP8WNC0Xc+iXu/BsnbRbaemDgcM693DMnSDhT2nG+rZXVtfWNzdJWeXtnd2/frhy0VZxKQlsk5rHsBlhRziLa0kxz2k0kxSLgtBOMr3O/80ClYnF0rycJ9QQeRixkBGsj+XalL7AeSZHdUDL1MzWa+nbVqTkzoGXiFqQKBZq+/dUfxCQVNNKEY6V6rpNoL8NSM8LptNxPFU0wGeMh7RkaYUGVl82iT9GJUQYojKV5kUYz9fdGhoVSExGYyTyoWvRy8T+vl+rw0stYlKSaRmR+KEw50jHKe0ADJinRfGIIJpKZrIiMsMREm7bKpgR38cvLpF2vuWe1+t15tXFV1FGCIziGU3DhAhpwC01oAYFHeIZXeLOerBfr3fqYj65Yxc4h/IH1+QP2CpR0</latexit> LCE : Enc p 0 Enc sh Dec sh Dec p 1 ; Enc p 1 Enc sh Dec sh Dec p 0 <latexit sha1_base64="JHovl8MvHOXmQNNtcZyC3i/oN/U=">AAACpHicnVHLSgMxFM2M7/qqunQTLYqrMqOCohtRiy66aNFqoa1DJr21wcyD5I5Yhvky/8Kdf2M6FtS2Ky8EDufce3IffiyFRsf5tOyZ2bn5hcWlwvLK6tp6cWPzQUeJ4tDgkYxU02capAihgQIlNGMFLPAlPPovV0P98RWUFlF4j4MYOgF7DkVPcIaG8orv7YBhnzOZVjMvvapkbYQ3TM9oRnNFBWkl5EaKPWec0v0f5hqmM7HnGir3pPScGtvCuK/7P18n84olp+zkQSeBOwIlMoqaV/xodyOeBBAil0zrluvE2EmZQsElZIV2oiFm/IU9Q8vAkAWgO2m+5IzuGaZLe5EyL0Sas78rUhZoPQh8kznsVI9rQ3Ka1kqwd9pJRRgnCGb+/KNeIilGdHgx2hUKOMqBAYwrYXqlvM8U42juWjBLcMdHngQPh2X3qHxYPy5dXI7WsUi2yS45IC45IRfkltRIg3Brx7qxalbd3rer9p3d+E61rVHNFvkT9tMXzpPTYg==</latexit> LBT : Enc p 1 Enc sh Dec sh Dec p 0 ; Enc p 0 Enc sh Dec sh Dec p 1 <latexit sha1_base64="oGvgJyH7EuFegnvsW5XKARpKPtA=">AAACpHicnVHJSgNBEO0Z97hFPXppDYqnMKOCohdxQQ85JGhMIIlDT6diGnsWumvEMMyX+Rfe/Bs7Y0BNcrKg4fFe1eta/FgKjY7zadkzs3PzC4tLheWV1bX14sbmo44SxaHOIxmpps80SBFCHQVKaMYKWOBLaPgvV0O98QpKiyh8wEEMnYA9h6InOENDecX3dsCwz5lMK5mXXj5kbYQ3TM9oRnNFBelNyI0Ue+44pfs/zDVMZ2LPMVTuSek5NbaFcV/nf75u5hVLTtnJg04CdwRKZBRVr/jR7kY8CSBELpnWLdeJsZMyhYJLyArtREPM+At7hpaBIQtAd9J8yRndM0yX9iJlXog0Z39XpCzQehD4JnPYqR7XhuQ0rZVg77STijBOEMz8+Ue9RFKM6PBitCsUcJQDAxhXwvRKeZ8pxtHctWCW4I6PPAkeD8vuUfmwdly6uBytY5Fsk11yQFxyQi7IHamSOuHWjnVrVa2avW9X7Hu7/p1qW6OaLfIn7Kcv7TLTcA==</latexit> Enc p 1 <latexit sha1_base64="GA5koI/FKjTrLss2eAQ6EpqabjY=">AAAB+3icbVDLSsNAFL3xWeur1qWbwSK4KkkVdFkUwWUF+4A2hMl00g6dScLMRCwhv+LGhSJu/RF3/o2TmIW2Hhg4nHMvc+7xY86Utu0va2V1bX1js7JV3d7Z3duvHdR7KkokoV0S8UgOfKwoZyHtaqY5HcSSYuFz2vdn17nff6BSsSi81/OYugJPQhYwgrWRvFp9JLCeSpHehCTz0thzMq/WsJt2AbRMnJI0oETHq32OxhFJBA014VipoWPH2k2x1IxwmlVHiaIxJjM8oUNDQyyoctMie4ZOjDJGQSTNCzUq1N8bKRZKzYVvJvOkatHLxf+8YaKDSzdlYZxoam4rPgoSjnSE8iLQmElKNJ8bgolkJisiUywx0aauqinBWTx5mfRaTees2bo7b7SvyjoqcATHcAoOXEAbbqEDXSDwCE/wAq9WZj1bb9b7z+iKVe4cwh9YH99pfZSt</latexit> Dec p 0 <latexit sha1_base64="UbcaR71z26AraQJR/yuM5MkW+cw=">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUmqoMuiLlxWsA9oQ5hMp+3QmSTMTMQS8ituXCji1h9x5984abPQ1gMDh3Pu5Z45QcyZ0o7zbZXW1jc2t8rblZ3dvf0D+7DaUVEiCW2TiEeyF2BFOQtpWzPNaS+WFIuA024wvcn97iOVikXhg57F1BN4HLIRI1gbyberA4H1RIr0lpLMT2PfyXy75tSdOdAqcQtSgwIt3/4aDCOSCBpqwrFSfdeJtZdiqRnhNKsMEkVjTKZ4TPuGhlhQ5aXz7Bk6NcoQjSJpXqjRXP29kWKh1EwEZjJPqpa9XPzP6yd6dOWlLIwTTUOyODRKONIRyotAQyYp0XxmCCaSmayITLDERJu6KqYEd/nLq6TTqLvn9cb9Ra15XdRRhmM4gTNw4RKacActaAOBJ3iGV3izMuvFerc+FqMlq9g5gj+wPn8AWH+Uog==</latexit> Dec p 1 <latexit sha1_base64="39OkXS3psyaEzLAtbnzdaTMAHLM=">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUmqoMuiLlxWsA9oQ5hMp+3QmSTMTMQS8ituXCji1h9x5984abPQ1gMDh3Pu5Z45QcyZ0o7zbZXW1jc2t8rblZ3dvf0D+7DaUVEiCW2TiEeyF2BFOQtpWzPNaS+WFIuA024wvcn97iOVikXhg57F1BN4HLIRI1gbyberA4H1RIr0lpLMT2PfzXy75tSdOdAqcQtSgwIt3/4aDCOSCBpqwrFSfdeJtZdiqRnhNKsMEkVjTKZ4TPuGhlhQ5aXz7Bk6NcoQjSJpXqjRXP29kWKh1EwEZjJPqpa9XPzP6yd6dOWlLIwTTUOyODRKONIRyotAQyYp0XxmCCaSmayITLDERJu6KqYEd/nLq6TTqLvn9cb9Ra15XdRRhmM4gTNw4RKacActaAOBJ3iGV3izMuvFerc+FqMlq9g5gj+wPn8AWgSUow==</latexit> Hi Hi En/CS En/CS LDAE : Enc p 0 Enc sh Dec sh Dec p 0 ; Enc p 1 Enc sh Dec sh Dec p 1 <latexit sha1_base64="7BRF3mozryImVTTS4MDCqnCuZpI=">AAACpXicnVHLSgMxFM2Mr1pfVZdugkV0VWZUUHRTtYILBSttFdoyZNLbNjTzILkjlmH+zK9w59+YjgW1uvJC4HDOvSf34cdSaHScd8uem19YXCosF1dW19Y3SptbLR0likOTRzJSTz7TIEUITRQo4SlWwAJfwqM/uproj8+gtIjCBo5j6AZsEIq+4AwN5ZVeOwHDIWcyvc28tHZxnXUQXjA9oxnNJRWk1yE3Wuw5s5QefjE1+Jv5LMs9KT2nxrY46+v+z9fNvFLZqTh50N/AnYIymca9V3rr9CKeBBAil0zrtuvE2E2ZQsElZMVOoiFmfMQG0DYwZAHobppvOaN7hunRfqTMC5Hm7PeKlAVajwPfZE461bPahPxLayfYP+2mIowTBDN//lE/kRQjOjkZ7QkFHOXYAMaVML1SPmSKcTSHLZoluLMj/watw4p7VDmsH5erl9N1FMgO2SUHxCUnpEpuyD1pEm7tWjdW3Xqw9+07u2G3PlNta1qzTX6E7X0AivvTrg==</latexit> Figure 1: Model architecture. (III) Cross-entropy loss (CE): Both the previous steps used unsupervised training objectives and make use of non-parallel text. With access to parallel text, one can use the standard supervised cross-entropy loss (denoted by LCE ) to train the translation models (i.e. going from Enc p 0 to Dec p 1 and Enc p 1 to Dec p 0 via the common shared layers). Apart from the use of parallel text and monolingual text employed in training TCS, we also construct large volumes of synthetic CS text using two simple techniques. This synthetic CS text is nonparallel and is used to optimize both LDAE and LBT . The role of the synthetic CS text is to expose TCS to various CS patterns (even if noisy), thereby encouraging the model to code-switch. The final step of finetuning using All-CS enables model to mimic switching patterns of real CS texts The first technique (named LEX) is a simple heuristic-based technique that constructs a CS sentence by traversing a Hindi sentence and randomly replacing a word by its English translation using a bilingual lexicon (Conneau et al., 2017). The probability of replacing a word is chosen to match the switching distribution in real CS text. The second technique (named EMT) is more linguistically aware. Following the methodology proposed by Bhat et al. (2016) that is based on the embedded 3157 matrix theory (EMT) for code-switching, we apply clause substitution methods to monolingual text to construct synthetic CS text. From inspecting English parse trees, we found that replacing embedded sentence clauses or subordinate clauses with their Hindi translations would likely produce CS text that appears somewhat natural. We introduce a new Hindi-English CS dataset, that we will refer to as All-CS. It is partitioned into two subsets, Movie-CS and Treebank-CS, based on their respective sources. Movie-CS consists of conversational Hindi-English CS text extracted from 30 contemporary Bollywood scripts that were publicly available. 3 The Hindi words in these sentences were all Romanized with potentially multiple non-canonical forms existing for the same Hindi token. We employed a professional annotation company to convert the Romanized Hindi words into their respective back-transliterated forms rendered in Devanagari script. We also asked the annotators to provide monolingual Hindi translations for all these sentences. Using these monolingual Hindi sentences as a starting point, we additionally crowdsourced for CS sentences via Amazon's Mechanical Turk (MTurk) (Amazon, 2005). Table 1 shows two Hindi sentences from Movie-CS and Treebank-CS, along with the different variants of CS sentences. Turkers were asked to convert a monolingual Hindi sentence into a natural-sounding CS variant that was semantically identical. Each Turker had to work on five Hindi sentences. We developed a web interface using which Turkers could easily copy parts of the Hindi sentence they wanted to retain and splice in English segments. More details about this interface, the crowdsourcing task and worker statistics are available in Appendix A. All-CS comprises a second subset of CS sentences, Treebank-CS, that was crowdsourcing using MTurk. We extracted 5292 monolingual Hindi sentences (with sentence lengths less than or equal to 15 words) from the publicly available Hindi Dependency Treebank that contains dependency parses. 4 These annotations parse each Hindi sentence into chunks, where a chunk is defined as 3 https://www.filmcompanion.in/category/fc-pro/scripts/ https://moifightclub.com/category/scripts/ 4 http://ltrc.iiit.ac.in/treebank_H2014/ Movie-CS   (cid:465)     (cid:465)   (Eng) (But laughter medicine really changed my life) (Gold) but laughter therapy  (cid:547) life  (cid:547) actually MTurk  laughter therapy     (cid:465)   MTurk but laughter therapy  really  (cid:547) life change  (cid:547) MTurk   therapy   life  (cid:465)   Treebank-CS    7.20   (cid:551)  (Eng) (Income from the fair was estimated at Rs 7.20 crore) MTurk fair  income 7.20   evaluate (cid:551)  MTurk   income 7.20   (cid:551)  Table 1: Two All-CS examples. a minimal, non recursive phrase. Turkers were asked to convert at least one Hindi chunk into English. This was done in an attempt to elicit longer spans of English segments within each sentence. Figure 2 shows the sentence length distributions for Movie-CS and Treebank-CS, along with histograms accumulating English segments of different lengths in both subsets. We clearly see a larger fraction of English segments with lengths within the range [2-6] in Treebank-CS compared to Movie-CS. Table 2 provides detailed statistics of the new CS dataset. We also report two metrics proposed by Guzmn et al. (2017) to measure the amount of code-switching present in this new corpus. Monolingual Index (M-Index) is a value between 0 and 1 Quantity/Metric Movie-CS Treebank-CS All-CS |Train| 15509 5914 21423 |Test| 1500 1000 2500 |Valid| 500 500 1000 # Tokens 196300 87979 284279 # Hindi Sentences 9290 5292 14582 # NEs 4342 4810 9152 Fraction of NEs 0.0221 0.0547 0.0322 M-Index 0.5542 0.6311 0.5774 I-Index 0.2852 0.3434 0.3023 Table 2: Key statistics of CS datasets. that quantifies the amount of mixing between languages (0 denotes a purely monolingual corpus and 1 denotes equal mixing from both languages) and I-Index measures the fraction of switching points in the corpus. We observe Treebank-CS exhibits higher M-index and I-index values compared to Movie-CS indicating more code-switching overall. All-CS also contains a non-trivial number of named entities (NEs) which are replaced by an NE tag in all our language modeling experiments. Parallel Hindi-English Text. As described in Section 5, TCS uses parallel text for supervised training. For this purpose, we use the IIT Bombay English-Hindi Corpus (Kunchukuttan et al., 2017) containing parallel Hindi-English text. We also construct a larger parallel corpus using text from the OpenSubtitles (OpSub) corpus (Lison and Tiedemann, 2016) that is more conversational and hence more similar in style to Movie-CS. We chose ~1 million English sentences (OpSub-EN), where each sentence contained an embedded clause or a subordinate clause to support the construction of EMT lines. We used the Google Translate API to obtain Hindi translations for all these sentences (OpSub-HI). Henceforth, we use OpSub to refer to this parallel corpus of OpSub-EN paired with OpSub-HI. We extracted 318K sentences from the IITB corpus after thresholding on length (5-15) and considering overlap in vocabulary with OpSub. (One could avoid the use of an external service like Google Translate and use existing parallel text (Zhang et al., 2020)) in conjunction with a word aligner to construct EMT lines. OpSub, being more conversational in style, turns out to be a better pretraining corpus. A detailed comparison of these choices is described in Appendix H.) Synthetic CS Datasets. As mentioned in Section 3.1, we use two simple techniques LEX and EMT to generate synthetic CS text, which in turn is used to train TCS in an unsupervised training phase. For each Hindi monolingual sentence in OpSub, we generate two LEX and two EMT synthetic CS sentences giving us OpSub-LEX and OpSub-EMT, respectively. We also generate five LEX and five EMT lines for each monolingual sentence in All-CS. In order to generate EMT lines, we first translate the monolingual Hindi sentences in All-CS to English using Google Translate and then follow the EMT generation scheme. This results in two datasets, All-CS-LEX and All-CS-EMT, which appear in later evaluations. (Appendix B contains more details about EMT applied to OPUS and All-CS.) Datasets from existing approaches. (I) VACS (Samanta et al., 2019) is a hierarchical variational autoencoder-based model designed to generate CS text. We train two VACS models, one on All-CS (VACSv1) and the other on OpSub-EMT followed by All-CS (VACSv2). (II) Garg et al. (2018a) use SeqGAN (Yu et al., 2017)  a GAN-based sequence generation model  to generate CS sentences by providing an RNNLM as the generator. As with VACS, we train two SeqGAN 5 models, one on All-CS (SeqGANv1) and one on OpSub-EMT followed by All-CS (SeqGANv2). Samples are drawn from both SeqGAN and VACS by first drawing a random sample from the standard normal distribution in the learned latent space and then decoding via an RNN-based generator for SeqGAN and a VAE-based decoder for VACS. We sample ~2M lines for each dataset to match the size of the other synthetic datasets. First, we investigate various training curricula to train TCS and identify the best training strategy by evaluating BLEU scores on the test set of All-CS (5.1). Next, we compare the output from TCS with synthetic CS text generated by other methods ( 5.2). We approach this via language modeling ( 5.2.1), human evaluations (5.2.2) and two downstream tasksNatural Language Inference and Sentiment Analysisinvolving real CS text (5.2.3). Apart from these tasks, we also present four different objective evaluation metrics to evaluate synthetic CS text: BERTScore, Accuracy of a BERT-based classifier and two diversity scores (5.3). Table 3 shows the importance of various training curricula in training TCS; these models are evaluated using BLEU (Papineni et al., 2002) scores computed with the ground-truth CS sentences for 5 https://github.com/suragnair/seqGAN the test set of All-CS. We start with supervised pretraining of TCS using the two parallel datasets we have in hand  IITB and OpSub (System A ). A is then further finetuned with real CS text in All-CS. The improvements in BLEU scores moving from System O (trained only on All-CS) to System B illustrate the benefits of pretraining TCS using Hindi-English parallel text. Systems C and D in Table 3 use our synthetic CS datasets OpSub-LEX and OpSub-EMT, respectively. These systems are further finetuned on All-CS using both unsupervised and supervised training objectives to give C 1 , C 2 , D 1 and D 2 , respectively. Comparing these four systems with System B shows the importance of using synthetic CS for pretraining. Further, comparing C 1 against D 1 and Figure 3: Variation of BLEU score with amount of All-CS parallel training data. C 2 against D 2 , we observe that OpSub-EMT is indeed a better choice for pretraining compared to OpSub-LEX. Also, supervised finetuning with All-CS is clearly superior to unsupervised finetuning. Henceforth, Systems D 1 and D 2 will be referred to as TCS (U) and TCS (S), respectively. While having access to parallel CS data is an advantage, we argue that the benefits of having parallel data only marginally increase after a threshold. Figure 3 shows how BLEU scores vary when changing the amount of parallel CS text used to train D 2 . We observe that BLEU increases substantially when we increase CS data from 1000 lines to 5000 lines, after which there is a trend of diminishing returns. We also find that D 1 (that uses the data in All-CS as non-parallel text) is as good as the model trained using 4000 lines of parallel text. We use text generated by our model to train a language model (LM) and evaluate perplexities on the test set of All-CS to show how closely sentences from TCS mimic real CS text. We use a state-of-the-art RNNLM model AWD-LSTM-LM Merity et al. (2018) as a blackbox LM and only experiment with different training datasets. The model uses three LSTM layers of 1200 hidden units with weight tying and 300-dimensional word embeddings. In initial runs, we trained our language model on the large parallel/synthetic CS datasets and finetuned on the All-CS data. However, this training strategy was prone to over-fitting on All-CS data. To counter this problem of forgetting during the pretrain-finetuning steps, we adopted the Mix-review strategy proposed by He et al. (2021). The training sentences from All-CS remain constant through the epochs and the amount of pretraining data is exponentially decayed with each epoch. This greatly alleviates the forgetting problem in our model, and leads to better overall perplexities. Additional details about these LMs are provided in Appendix E. Table 4 shows test perplexities using different training curricula and data generated using two prior approaches, VACS and SeqGAN. Sentences generated using TCS yield the largest reductions in test perplexities, compared to all other approaches. We evaluated the quality of sentences generated by TCS using a human evaluation study. We sampled 150 sentences each, using both TCS (U) and TCS (S), starting from monolingual Hindi sentences in the evaluation sets of All-CS. The sentences were chosen such that they were consistent with the length distribution of All-CS. For the sake of comparison, corresponding to the above-mentioned 150 monolingual Hindi samples, we also chose 150 CS sentences each from All-CS-LEX and All-CS-EMT. Along with the ground-truth CS sentences from All-CS, this resulted in a total of 750 sentences. 6 These sentences were given to three linguistic experts in Hindi and they were asked to provide scores ranging between 1 and 5 (1 for worst, 5 for best) under three heads: Syntactic correctness, Semantic correctness and Natu-ralness. Table 5 shows that the sentences generated using TCS (S) and TCS (U) are far superior to the EMT and LEX sentences on all three criteria. TCS (S) is quite close in overall quality to the real sentences and TCS (U) fares worse, but only by a small margin. 6 We only chose CS sentences from TCS that did not exactly match the ground-truth CS text. We observe that the model is able to introduce long contiguous spans of English words (e.g. meeting next week, but it is clear, etc.). The model also displays the ability to meaningfully switch multiple times within the same sentence (e.g., i love you very much, but, friend). There are also interesting cases of English segments that appear to be ungrammatical but make sense in the CS context (e.g., because i know main dish, etc.). GLUECoS (Khanuja et al., 2020) is an evaluation benchmark spanning six natural language tasks for code-switched English-Hindi and English-Spanish data. The authors observe that M-BERT (Pires et al., 2019) consistently outperforms cross-lingual embedding techniques. Furthermore, pretraining M-BERT on small amounts of code-switched text improves its performance in most cases. For our evaluation, we select two tasks that require semantic understanding: Natural Language Inference (NLI) and Sentiment Analysis (SA). We sample 100K monolingual sentences from 3161 Pretraining Data NLI (Accuracy) SentimentAnalysis(F1) Baseline 57.88  1.22 57.97  0.06 OpSub-HI 58.47  0.36 58.13  0.25 OpSub-LEX 58.67  0.94 58.40  0.33 OpSub-EMT 58.96  0.70 58.79  0.37 TCS (S) 59.57  0.57 59.39  0.81 All-CS 59.74  0.96 58.77  0.44 Table 7: GLUECoS Evaluation: Mean and standard deviation of scores after evaluating on 5 seeds. When subject to examples from high-quality generators, the classifier should find it hard to tell apart real from fake Evaluation Metric Real LEX EMT TCS (S) TCS (U) BERTScore All (3500) 0.812 0.796 0.627 0.764 0.788 Mono (3434) 0.812 0.782 0.623 0.755 0.772 UNK (1983) 0.809 0.804 0.636 0.827 0.846 UNK & Mono (1857) 0.808 0.785 0.633 0.813 0.821 BERT-based Classifier | Sentences | 4767 12393 12484 12475 12475 Accuracy(fake) 42.76 96.52 97.83 80.31 88.62 Diversity Gzip ( D ) 22.13 24.12 33.17 21.37 17.59 Self-BLEU 61.3 29.7 24.6 63.6 64.2 Table 8: (a) BERTScores on test split of All-CS. Each row corresponds to a different data filter. The numbers in parenthesis denote the number of sentences in the data after filtering. (b) Accuracies from the classifier for samples generated by various methods as being fake. The | Sentences | refer to size of dataset for each system. TCS models have the lowest accuracy among synthetic methods. (c) Diversity Scores for different techniques using Gzip and Self-BLEU based diversity measures. OpSub-HI and select corresponding LEX, EMT and TCS (S) sentences. M-BERT is then trained using the masked language modelling (MLM) objective on text from all 4 systems (including OpSub-HI) for 2 epochs. We also train M-BERT on 21K sentences from All-CS (real CS). Finally, these pretrained models are fine-tuned on the selected GLUECoS tasks. (More details are in Appendix G.) Table 7 lists the accuracies and F1 scores using different pretraining schemes for both NLI and sentiment analysis, respectively. Plain monolingual pretraining by itself leads to performance improvements on both tasks, presumably due to do-main similarity between GLUECoS (movie scripts, social media etc.) and OpSub. As mentioned in Khanuja et al. (2020), pretraining on CS text further improves performance for both NLI and SA. Among the synthetic methods, TCS (S) has consistently better scores than LEX and EMT. For SA, TCS (S) even outperforms pretraining on real CS text from All-CS. BERTScore. BERTScore (Zhang* et al., 2020) is a recently-proposed evaluation metric for text generation. Similarity scores are computed between each token in the candidate sentence and each token in the reference sentence, using contextual BERT embeddings (Devlin et al., 2018) of the tokens. We use this as an additional objective metric to evaluate the quality of the sentences generated using TCS. We use the real monolingual sentence as the reference and the generated CS sentence as the candidate, excluding sentences from TCS (S) and TCS (U) that exactly match the real sentence. Since our data is Hindi-English CS text, we use Multilingual BERT (M-BERT) (Pires et al., 2019) for high-quality multilingual representations. Table 8 outlines our main results on the test set of All-CS. TCS sometimes generates purely monolingual sentences. This might unfairly tilt the scores in favour of TCS since the reference sentences are also monolingual. To discount for such biases, we remove sentences generated by TCS (U) and TCS (S) that are purely monolingual (Row label Mono in BERTScore). Sentences having <UNK> tokens (labeled UNK) are also filtered out since these tokens are only generated by TCS for out-of-vocabulary words. UNK & Mono refers to applying both these filters. EMT lines consistently show the worst performance, which is primarily due to the somewhat poor quality of translations involved in generating these lines (refer to Appendix B). With removing both monolingual and <UNK> tokens, we observe that TCS (U) and TCS (S) yield the highest BERTScores, even outperforming the BERTScore on real data obtained from the Turkers. BERT-based Classifier. In this evaluation, we use M-BERT ( Pires et al., 2019) to build a classifier that distinguishes real CS sentences from synthetically generated ones (fake). samples. We add a fully connected layer over the M-BERT base architecture that takes the [CLS] token as its input to predict the probability of the sentence being real or fake. Fake sentences are drawn from the union of TCS (U), TCS (S), All-CS-LEX and All-CS-EMT. In order to alleviate the class imbalance problem, we oversample the real sentences by a factor of 5 and shuffle the data. The model converges after training for 5 epochs. We see in Table 8 that the classification accuracy of whether a sample is fake or not is lowest for the outputs from TCS among the different generation techniques. Measuring Diversity. We are interested in finding out how diverse the predictions from TCS are. We propose a simple measure of diversity in the CS variants that is based on how effectively sentences can be compressed using the gzip utility. 7 We considered using Byte Pair Encoding (BPE) (Gage, 1994) as a measure of data compression. However, BPE operates at the level of individual words. Two word sequences w1 w2 w3 and w3 w2 w1 would be identically compressed by a BPE to-kenizer. We would ideally like to account for such diversity and not discard this information. gzip uses Lempel-Ziv coding (Ziv and Lempel, 1977) that considers substrings of characters during compression, thus allowing for diversity in word ordering to be captured. Our diversity measure D is simply the following: For a given set of CS sentences, run gzip on each sentence individually and sum the resulting file sizes ( S 1 ). Next, paste all the CS sentences into a single file and run gzip on it to get a file of size S 2 . Then, D = S 1  S 2 . Smaller D scores indicate larger diversity. If the variants of a sentence are dissimilar to one another and hence very diverse, then S 2 would be large thus leading to smaller values of D . Table 8 shows the diversity scores for different techniques. Both TCS (S) and TCS (U) have a higher diversity score compared to LEX and EMT. TCS (U) exceeds even the responses received via MTurk (Real) in diversity. We note here that diversity, by itself, is not necessarily a desirable trait. Our goal is to generate sentences that are diverse while being natural and semantically meaningful. The latter properties for text from TCS (S) and TCS (U) have already been verified in our human evaluation study. However, using self-BLEU is slightly problematic in our setting as systems like LEX that switch words at random positions would result in low self-BLEU (indicating high diversity). This is indeed the case, as shown in Table 8 LEX, EMT give lower self-BLEU scores as compared to TCS. However, note that the scores of the TCS models are comparable to that of real CS data. In this work, we present a neural translation model for CS text that transduces monolingual Hindi sentences into realistic Hindi-English CS text. Text generated by our model is evaluated using a number of different objective metrics, along with LM, NLI and sentiment analysis tasks, and a detailed human evaluation study. The role of synthetic data in training such models merits a more detailed investigation which we leave for future work. We thank all the anonymous reviewers for their constructive feedback which helped improve the presentation of this work. We also thank all the volunteers who helped with the collection of CS text that is released as part of our dataset, All-CS.
abstain method method result result objective abstain abstain abstain abstain abstain abstain method method method method method result method abstain abstain method objective objective objective objective method objective result objective other other other other other other other other other other other method method other method method objective other objective method abstain abstain abstain method method method method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain method method abstain method method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain method objective method other other