We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
This paper shows that an LSTM language model trained on a large corpus of Amazon product reviews can learn representations that are useful for sentiment analysis. 
Given representations from the language model, a logistic regression classifier is trained with supervised data from the task of interest to produce the final model.
The authors evaluated their approach on six sentiment analysis datasets (MR, CR, SUBJ, MPQA, SST, and IMDB), and found that the proposed method is competitive with existing supervised methods. 
The results are mixed, and they understandably are better for test datasets from similar domains to the Amazon product reviews dataset used to train the language model.
An interesting finding is that one of the neurons captures sentiment property and can be used to predict sentiment as a single unit.
I think the main result of the paper is not surprising and does not show much beyond we can do pretraining on unlabeled datasets from a similar domain to the domain of interest. 
This semi-supervised approach has been known to improve in the low data regime, and pretraining an expressive neural network model with a lot of unlabeled data has also been shown to help in the past.
There are a few unanswered questions in the paper:
- What are the performance of the sentiment unit on other datasets (e.g., SST, MR, CR)? Is it also competitive with the full model?
- How does this method compare to an approach that first pretrains a language model on the training set of each corpus without using the labels, and then trains a logistic regression while fixing the language model? Is the large amount of unlabeled data important to obtain good performance here? Or is similarity to the corpus of interest more important?
- I assume that the reason to use byte LSTM is because it is cheaper than a word level LSTM. Is this correct or was there any performance issue with using the word directly?
- More analysis on why the proposed method does well on the binary classification task of SST, but performs poorly on the fine-grained classification would be useful. If the model is capturing sentiment as is claimed by the authors, why does it only capture binary sentiment instead of a spectrum of sentiment level?
The paper is also poorly written. There are many typos (e.g., "This advantage is also its difficulty", "Much previous work on language modeling has evaluated ", "We focus in on the task", and others) so the writing needs to be significantly improved for it to be a conference paper, preferably with some help from a native English speaker.