Deconvolutional Paragraph Representation Learning
Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the length of the text. We propose a sequence-to-sequence, purely convolutional and deconvolutional autoencoding framework that is free of the above issue, while also being computationally efficient. The proposed method is simple, easy to implement and can be leveraged as a building block for many applications. We show empirically that compared to RNNs, our framework is better at reconstructing and correcting long paragraphs. Quantitative evaluation on semi-supervised text classification and summarization tasks demonstrate the potential for better utilization of long unlabeled text data.
This paper proposes encoder-decoder models that uses convolutional neural networks, not RNNs. Experimental results on NLP data sets show that the proposed models outperform several existing methods.
In NLP, RNNs are more commonly used than CNNs. This is because the length of an input varies. CNNs can only handle a fixed length of an input, by truncating input text or padding shortage of the input. In experiments, it is okay to assume the length of an input, such as 50-250 words; however, in reality, we need to handle an arbitrary length of text information, sometimes 10 and sometimes 1,000 words. In this respect, RNNs, especially, LSTMs, have high potential in NLP applications.
Major Comments:
1. For summarization experiments, Document Understanding Conference (DUC) data sets and CNN news corpus are commonly used.
2. For the headline summarization, if you use shared data, we can judge how your models are better than other methods, including non-neural approaches.
3. Table 2 shows that the CNN-DCNN model can directly output an input. Extremely high BLEU scores imply low generalization. It has a sufficient rephrasing ability, it cannot achieve the BLUE scores over 90. 
4. In image processing, similar models have already been proposed, such as:
V. Badrinarayanan, A. Handa, and R. Cipolla. SegNet: a deep convolutional encoder-decoder architecture
for robust semantic pixel-wise labelling. arXiv preprint
arXiv:1505.07293, 2015.
Please make your contributions clear in your paper.
5. I am not sure how your semi-supervised experiments have been conducted. In what sense, your methods are semi-supervised?
6. Regarding the compared LSTM-LSTM encoder-decoder model, is it an attention-based encoder-decoder model?
7. I suggest to try machine translation data sets, in your future work.