This article proposes to auto-encode text at byte-level using convolutional networks with a recursive architecture. The motivation is to explore whether it is possible to have scalable and homogeneous text generation at byte-level in a nonsequential fashion through the simple task of auto-encoding. We show that nonsequential text generation from a fixed-length representation is not only possible, but also achieved much better auto-encoding results than recurrent networks. The proposed model is a multi-stage deep convolutional encoder-decoder framework using residual connections (He et al., 2016), containing up to 160 parameterized layers. Each encoder or decoder contains a shared group of modules that consists of either pooling or upsampling layers, making the network recursive in terms of abstraction levels in representation. Results for 6 large-scale paragraph datasets are reported, in 3 languages including Arabic, Chinese and English. Analyses are conducted to study several properties of the proposed model.
Recently, generating text using convolutional networks (ConvNets) starts to become an alternative to recurrent networks for sequence-tosequence learning (Gehring et al., 2017). The dominant assumption for both these approaches is that texts are generated one word at a time. Such sequential generation process bears the risk of output or gradient vanishing or exploding problem (Bengio et al., 1994), which limits the length of its generated results. Such limitation in scalability prompts us to explore whether non-sequential text generation is possible.
Meanwhile, text processing from lower levels than words -such as characters (Zhang et al., 2015)  (Kim et al., 2016) and bytes (Gillick et al., 2016) (Zhang & LeCun, 2017) -is also being explored due to its promise in handling distinct languages in the same fashion. In particular, the work by Zhang & LeCun (2017) shows that simple one-hot encoding on bytes could give the best results for text classification in a variety of languages. The reason is that it achieved the best balance between computational performance and classification accuracy. Inspired by these results, this article explores auto-encoding for text using byte-level convolutional networks that has a recursive structure, as a first step towards low-level and non-sequential text generation.
For the task of text auto-encoding, we should avoid the use of common attention mechanisms like those used in machine translation Bahdanau et al. (2015), because they always provide a direct information path that enables the auto-encoder to directly copy from the input. This diminishes the Figure 2: Sequential and non-sequential decoders illustrated in graphical models. u is a vector containing encoded representation. y i 's are output entities. h i 's are hidden representations. Note that they both imply conditional independence between outputs conditioned by the representation u.
purpose of studying the representational ability of different models. Therefore, all models considered in this article would encode to and decode from a fixed-length vector representation.
The paper by  was an anterior result on using word-level convolutional networks for text auto-encoding. This article differs from it in several key ways of using convolutional networks. First of all, our models work from the level of bytes instead of words, which arguably makes the problem more challenging. Secondly, our network is dynamic with a recursive structure that scales with the length of input text, which by design could avoid trivial solutions for auto-encoding such as the identity function. Thirdly, by using the latest design heuristics such as residual connections (He et al., 2016), our network can scale up to several hundred of layers deep, compared to a static network that contains a few layers.
In this article, several properties of the auto-encoding model are studied. The following is a list.
1. Applying the model to 3 languages -Arabic, Chinese and English -shows that the model can handle all different languages in the same fashion with equally good accuracy. 2. Comparisons with long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) show a significant advantage of using convolutional networks for text auto-encoding. 3. We determined that a recursive convolutional decoder like ours can accurately produce the end-of-string byte, despite that the decoding process is non-sequential. 4. By studying the auto-encoding error when the samples contain randomized noisy bytes, we show that the model does not degenerate to the identity function. However, it can neither denoise the input very well. 5. The recursive structure requires a pooling layer. We compared between average pooling, L2 pooling and max-pooling, and determined that max-pooling is the best choice. 6. The advantage of recursion is established by comparison against a static model that does not have shared module groups. This shows that linguistic heuristics such as recursion is useful for designing models for language processing. 7. We also explored models of different sizes by varying the maximum network depth from 40 to 320. The results show that deeper models give better results.
The paper aims to illustrated the representation learning ability of the convolutional autoencoder with residual connections is  proposed by to encode text at the byte level.  The authors apply the proposed architecture to 3 languages and run comparisons with an LSTM.  Experimental results  with different perturbation of samples, pooling layers, and sample lengths are presented.
The writing is fairly clear, however the presentation of tables and figures could be done better, for example, Fig. 2 is referred  to in page 3,  Table 2 which contains results is referred to on page 5, Fig 4 is referred to in page 6 and appears in page 5, etc.
What kind of minimal preprocessing is done on the text? Are punctuations removed? Is casing retained? How is the space character encoded?
Why was the encoded dimension always fixed at 1024?  What is the definition of a sample here?
The description of the various data sets could be moved to a table/Appendix, particularly since most of the results are presented on the enwiki dataset, which would lead to better readability of the paper.  Also results are presented only on a random 1M sample selected from these data sets, so the need for this whole page goes away.
Comparing Table 2 and Table 3, the LSTM is at 67% error on the test set while the proposed convolutional autoencoder is at 3.34%.  Are these numbers on the same test set?    While the argument that the LSTM does not generalize well due to the inherent memory learnt is reasonable, the differences in performance cannot be explained away with this. Can you please clarify this further?
It appears that the byte error shoot up for sequences of length 512+ (fig. 6 and fig. 7) and seems entirely correlated with the amount of data than recursion levels.
How do you expect these results to change for a different subset selection of training and test samples? Will Fig. 7 and Fig. 6 still hold?
In Fig, 8, unless the static train and test error are exactly on top of the  recursive errors, they are not visible.  What is the x-axis in Fig. 8?  Please also label axes on all figures. 
While the datasets are large and would take a lot of time to process for each case study, a final result on the complete data set, to illustrate if the model does learn well with lots of data would have been useful.  A table showing generated sample text would also clarify the power of the model.
With the results presented,  with a single parameter setting, its hard to determine what exactly the model learns and why.