While many sentiment classification solutions report high accuracy scores in product or movie review datasets, the performance of the methods in niche domains such as finance still largely falls behind. The reason of this gap is the domainspecific language, which decreases the applicability of existing models, and lack of quality labeled data to learn the new context of positive and negative in the specific domain. Transfer learning has been shown to be successful in adapting to new domains without large training data sets. In this paper, we explore the effectiveness of NLP transfer learning in financial sentiment classification. We introduce FinBERT, a language model based on BERT, which improved the state-of-the-art performance by 14 percentage points for a financial sentiment classification task in FinancialPhrasebank dataset.
This paper described the application of BERT in the field of financial sentiment analysis. Authors find that when fine-tuned with in-domain data, BERT outperforms the state-of-the-art, demonstrating that language model pre-training can transfer knowledge learned from unsupervised large corpus to new domain with minimum effort. Experiments are conducted to explore 1) the utility of different in-domain dataset for further pre-training; 2) strategies to avoid catastrophic forgetting, and 3) effectiveness of fine-tuning a subset of the full model. 
I am in favor of rejecting this paper and my reasons are as follows:
First, this paper may lack deeper innovation, although it demonstrates a good application of the BERT models in financial domain. For example, the framework of general-domain LM pretraining, to in-domain LM pretraining and finally in-domain classifier fine-tuning, as well as techniques of catastrophic forgetting were already proposed in Howard & Ruder 2018. Therefore, I think this paper may be more suitable for other (finance) application-oriented venues.
Second, the dataset used in evaluation is of small size (for example, Financial PhraseBank test set has one 1K). Thus, even though the paper is about transfer learning to domains without large data, I find it might be more convincing to draw a solid conclusion with a larger test set.
This paper is well organized and easy to follow. It may be beneficial to clarify in a few places (if space permits):
1) Some description or statistics of the data may be helpful (e.g., average sentence length or some examples);
2) Citations to Elmo and ULMFit can be made more explicit. Authors did cite Peters 2018 and Howard 2018 at the beginning of the paper, but may want to explicitly associate them with ‘Elmo’ and ‘ULMFit’ when these two terms first occur respectively;
3) For table 2, does the ‘all data’ or ‘data with 100% agreement’ include training data (80%) or just the test data (20%)?
The difference between FinBERT(-domain) and ULMFit can be explicitly contrasted in the paper. Is the former initialized with BERT while latter with ULMFit?