Which downstream tasks are used for evaluation in this paper?
Various sequence tagging tasks: Argument detection, ACE entity and event detection, part-of-speech tagging, CoNLL chunking, CoNLL named entity recognition, GENIA bio-entity recognition, WNUT named entity recognition. They also evaluate on Stanford Sentiment Treebank, Penn TreeBank constituency parsing, and Stanford Natural Language Inference.
ELMo embeddings (Peters et. al, 2018) had a huge impact on the NLP community and may recent publications use these embeddings to boost the performance for downstream NLP tasks. However, integration of ELMo embeddings in existent NLP architectures is not straightforward. In contrast to traditional word embeddings, like GloVe or word2vec embeddings, the bi-directional language model of ELMo produces three 1024 dimensional vectors per token in a sentence. Peters et al. proposed to learn a task-specific weighting of these three vectors for downstream tasks. However, this proposed weighting scheme is not feasible for certain tasks, and, as we will show, it does not necessarily yield optimal performance. We evaluate different methods that combine the three vectors from the language model in order to achieve the best possible performance in downstream NLP tasks. We notice that the third layer of the published language model often decreases the performance. By learning a weighted average of only the first two layers, we are able to improve the performance for many datasets. Due to the reduced complexity of the language model, we have a training speed-up of 19-44% for the downstream task.
Peters2018 presented in their work Deep Contextualized Word Representations (often referred to as ELMo embeddings) a method that uses a bidirectional language model (biLM) to derive word representations which are based on the complete context of a sentence. They demonstrated that these ELMo embeddings can substantially increase the performance for various NLP tasks. This new type of word representations had a big impact on the NLP community and many new architectures, for example, many from EMNLP 2018, report a better performance when using ELMo embeddings. Traditional word embeddings, like word2vec or GloVe, mapped each token in a sentence to a single dense vector. In contrast to that, the published ELMo implementation computes three layers of a language model: The first layer is a CNN that computes a non-contextualized word representation based on the characters of a word, followed by two bidirectional LSTM layers that take the context of the sentence into account. The output of the three layers is integrated into task-specific neural architectures. However, the integration of ELMo into neural architectures is not straightforward. For example, Peters et al. describe two methods for the integration: Either the output of the last layer is used for downstream tasks, or a task-specific weighting of the three layer outputs is learned: $\text{ELMo}_{\text{weighted\_average}} = \gamma \sum _{j=0}^{2}s_j h_j$  with $s \in \mathbb {R}^3$ softmax-normalized weights, $h_j$ the output of the $j$ -th layer of the biLM and $\gamma $ a scalar that is used to scale the entire ELMo vector. Learning this weighted average is not always easy, as it can require substantial changes in existent network architectures and some deep learning frameworks (for example Keras) lack the possibility to easily implement such a learned weighted average. Further, for unsupervised tasks, such a weighted average cannot be learned. Hence, several authors used simplified ways to integrate ELMo embeddings in their architectures. Some use the output of the last ELMo layer, some concatenate all three vectors BIBREF1 , BIBREF2 , and others compute a (fixed) average of the three layers. It is unclear what the impact of these different weighting schemes is for downstream tasks. Is the (rather complicated) learned weighted average proposed by Peters et al. needed to achieve optimal performance? Will a simpler method, like computing a fixed average, decrease the performance? In this paper, we evaluate different schemes to combine the three ELMo vectors. We analyze the impact of these schemes for downstream NLP tasks. First, we study this for a BiLSTM-CRF architecture which only uses ELMo embeddings as input representation. Next, we study the different weighting schemes for the more complex models included in AllenNLP, which concatenate ELMo embeddings with other input representations like GloVe word embeddings. In this paper we show that 1) the weighting scheme can have a significant impact on downstream NLP tasks, 2) that the learned weighted average proposed by Peters et al. does not yield the optimal performance for all datasets, and 3) that the second layer of the biLM yields in many cases a better performance than the third (last) layer. Surprisingly, using the output of the second layer of the biLM model yields a better performance than using the third (last) layer in many downstream NLP tasks. Using this insight, we present a weighting scheme that learns a weighted average of the first two layers of the biLM. This scheme outperforms the originally proposed weighting scheme by Peters et al. for several datasets. Further, it is computationally faster than the original method. For downstream tasks, we saw a training speed-up of 19-44%.