ABSTRACT Auto-regressive model architectures are the workhorse of sequence generation tasks in text, images, audio, and more. These models are predominantly trained with teacher-forcing, wherein the model is encouraged to predict the next ground-truth token conditioned on all previous ground-truth tokens in the sequence. At test time, the model does not have access to ground-truth tokens and must instead condition on its own prior predictions, resulting in a train-test discrepancy known as exposure bias (Ranzato et al., 2015). Scheduled Sampling (Bengio et al., 2015) aims to mitigate exposure bias by randomly replacing some tokens with model predictions during the course of training. Unfortunately, this technique is inherently sequential and unlike teacher-forcing, cannot be trivially parallelized across time. In this paper, we introduce a simple technique to parallelize Scheduled Sampling, significantly increasing training throughput. Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forcing. In dialog response generation, Parallel Scheduled Sampling achieves a 1.6 BLEU score (11.5%) improvement over teacher-forcing with more than 250 times higher throughput than Scheduled Sampling. In image generation, our approach achieves 20% and 13.8% improvement in Fréchet Inception Distance (FID) and Inception Score (IS) respectively. Finally, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.
Scheduled Sampling aims to overcome the problems that come with the discrepancy between train and test
when training sequence generation models using teacher-forcing. However one of the drawbacks scheduled sampling is that is it is hard to parallelize this is simply because some of the decode input tokens aren't known beforehand and need to be inferred i.e. loss of some tokens are dependent on some previous generations. 
Authors introduce a method to perform parallel scheduled sampling by iteratively train using teacher forcing however after each iteration the gold references are mixed with the model predictions with a probability p, this allows parallelization as all decoder input tokens can be known beforehand. The final loss is calculated as before, conditioning on the final mixture.
In theorem 2.1 Authors proof that parallel scheduled sampling can converge to scheduled sampling when P is set to 1 and the number of iterations is larger than the sentence length. 
In experiments over summarization, dialogue response generation and image generation authors show that the parallel scheduled sampling can achieve comparable performance to scheduled sampling with high-performance gains reaching 300 times faster and very comparable to teacher forcing. 
pros:
The work presents a simple idea that is presented neatly with sufficient experiments. One of the outcomes of this method, that might not been stressed enough in the paper, is a neat way of doing scheduled sampling for the transformers architecture which wasn't straight forward before. 
Current proposals include some architecture modifications such as in Mihaylova et al. 2019 https://arxiv.org/pdf/1906.07651.pdf
cons:
The only drawback I can find in this paper is it is lacking content. while the idea is interesting and supported by experiments, I find the content is slightly below the amount of content in average ICLR papers. 
The explanation of the proposed scheduled sampling can be much simplified. The idea is quite simple however for example I find the pseudo code in Algorithm 2 is quite hard to grasp maybe due to some typos
Questions to authors: 
- We only see performance comparison in dialogue response generation experiments. There are other factors that can affect performance or make parallelization effective.  I wonder what are the performance gains of parallel scheduled sampling on normal scheduled sampling with respect to avg. number of tokens / sentence or batch size. 
- I had a hard time grasping Algorithm 2 although I understood the corresponding text could you please verify there aren't any typos. What it says is in the first iteration (k=1) scheduled sampling will be performed which doesn't make sense.