This paper proposes a new model for document embedding. Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize. We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model. This allows us to train deeper architectures that are fully parallelizable. Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document. Empirically we demonstrate superior results on two publicly available benchmarks. Full code will be released with the final version of this paper.
This paper proposes a new model for the general task of inducing document representations (embeddings). The approach uses a CNN architecture, distinguishing it from the majority of prior efforts on this problem, which have tended to use RNNs. This affords obvious computational advantages, as training may be parallelized. 
Overall, the model presented is relatively simple (a good thing, in my view) and it indeed seems fast. I can thus see potential practical uses of this CNN based approach to document embedding in future work on language tasks. The training strategy, which entails selecting documents and then indexes within them stochastically, is also neat. Furthermore, the work is presented relatively clearly. That said, my main concerns regarding this paper are that: (1) there's not much new here, and, (2) the experimental setup may be flawed, in that it would seem model hyperparams were tuned for the proposed approach but not for the baselines; I elaborate on these concerns below.
Specific comments:
---
- It's hard to tease out exactly what's new here: the various elements used are all well known. But perhaps there is merit in putting the specific pieces together. Essentially, the novelty is using a CNN rather than an RNN to induce document embeddings. 
- In Section 4.1, the authors write that they report results for their after running "parameter sweeps ..." -- I presume that these were performed on a validation set, but the authors should say so. In any case, a very potential weakness here: were analagous parameter sweeps for this dataset performed for the baseline models? It would seem not, as the authors write "the IMDB training data using the default hyper-parameters" for skip-thought. Surely it is unfair comparison if one model has been tuned to a given dataset while others use only the default hyper-parameters? 
- Many important questions were left unaddressed in the experiments. For example, does one really need to use the gating mechanism borrowed from the Dauphin et al. paper? What happens if not? How big of an effect does the stochastic sampling of document indices have on the learned embeddings? Does the specific underlying CNN architecture affect results, and how much? None of these questions are explored. 
- I was left a bit confused regarding how the v_{1:i-1} embedding is actually estimated; I think the details here are insufficient in the current presentation. The authors write that this is a "function of all words up to w_{i-1}". This would seem to imply that at test time, prediction is not in fact parallelizable, no? Yet this seems to be one of the main arguments the authors make in favor of the model (in contrast to RNN based methods). In fact, I think the authors are proposing using the (aggregated) filter activation vectors (h^l(x)) in eq. 5, but for some reason this is not made explicit. 
Minor comments:
- In Eq. 4, should the product be element-wise to realize the desired gating (as per the Dauhpin paper)? This should be made explicit in the notation.
- On the bottom of page 3, the authors claim "Expanding the prediction to multiple words makes the problem more difficult since the only way to achieve that is by 'understanding' the preceding sequence." This claim should either by made more precise or removed. It is not clear exactly what is meant here, nor what evidence supports it.
- Commas are missing in a few. For example on page 2, probably want a comma after "in parallel" (before "significantly"); also after "parallelize" above "Approach".
- Page 4: "In contrast, our model addresses only requires" --> drop the "addresses".