A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning but through two separate channels corresponding to their languages. In this work, we present Multichannel Generative Language Models (MGLM), which models the joint distribution over multiple channels, and all its decompositions using a single neural network. MGLM can be trained by feeding it k way parallel-data, bilingual data, or monolingual data across pre-determined channels. MGLM is capable of both conditional generation and unconditional sampling. For conditional generation, the model is given a fully observed channel, and generates the k − 1 channels in parallel. In the case of machine translation, this is akin to giving it one source, and the model generates k − 1 targets. MGLM can also do partial conditional sampling, where the channels are seeded with prespecified words, and the model is asked to infill the rest. Finally, we can sample from MGLM unconditionally over all k channels. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads to improvements in bilingual translations. We provide a quantitative analysis of the quality-diversity trade-offs for different variants of the multichannel model for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.
[Paper summary]
This work is an extension of KERMIT (Chan et al., 2019) to multiple languages and the proposed model is called “multichannel generative language models”. KERMIT is an extension of “Insertion Transformer” (Stern et. al, 2019), a non-autoregressive model that can jointly determine which word and which place the translated words should be inserted. KERMIT shares the encoder and decoder of insertion Transformer, and the source sentence and target sentence are concatenated to train a generative model (also, various loss functions are included). In this work, parallel sentences from more than two languages are concatenated together and fed into KERMIT. Each language is associated with a language embedding. This work demonstrates that a joint distribution p(x1, . . . , xk) over k channels/languages can be properly modeled through a single model. The authors carry out experiments on multi30k dataset.
[Pros] Some discoveries of this work are interesting, including: (1) It is possible to use a single model to translate a sentence into different languages in a non-autoregressive way. (2) The unconditional multilingual generation in Section 4.5 is interesting, especially, the generation order is determined by the model rather than left-to-right.
[Questions]
1.	The authors work on multi30k dataset, which is not a typical dataset for machine translation. 
(A)	The dataset and the corresponding information is at https://github.com/multi30k/dataset. The number of words in a sentence is smaller than 15, which is too short for a machine translation. Also, the pattern of sentences is relatively simple.
(B)	For real world application, I am not sure whether it is possible to collect a large amount of k-parallel data where $k>2$. Therefore, the application scenario is limited. What if we have a large amount of bilingual data instead of k-parallel data? How should we leverage the large amount of monolingual data?
2.	For novelty, this is an extension of KERMIT to a multilingual version, which limits the novelty of this wok.
3.	The best results on En->De in Table 1 are inconsistent. On tst16, bilingual en<->de is the best; on tst17, en<->{rest} is the best; on mscoco, any<->rest is the best. In Table 2, seems using bilingual data only is the best choice. This makes me confuse about how to use your proposed method. However,