Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks. On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks. On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings. We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings. In this paper, we formulate the learning of word embeddings as a lifelong learning process. Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the metalearner is able to provide word context similarity information at the domain-level. Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledges
1 . We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks.
This paper presents a lifelong learning method for learning word embeddings.  Given a new domain of interest, the method leverages previously seen domains in order to hopefully generate better embeddings compared to ones computed over just the new domain, or standard pre-trained embeddings.
The general problem space here -- how to leverage embeddings across several domains in order to improve performance in a given domain -- is important and relevant to ICLR.  However, this submission needs to be improved in terms of clarity and its experiments.
In terms of clarity, the paper has a large number of typos (I list a few at the end of this review) and more significantly, at several points in the paper is hard to tell what exactly was done and why.  When presenting algorithms, starting with an English description of the high-level goal and steps of the algorithm would be helpful.  What are the inputs and outputs of the meta-learner, and how will it be used to obtain embeddings for the new domain?  The paper states the purpose of the meta learning is "to learn a general word context similarity from the first m domains", but I was never sure what this meant.  Further, some of the paper's pseudocode includes unexplained steps like "invert by domain index" and "scanco-occurrence".  
In terms of the experiments, the paper is missing some important baselines that would help us understand how well the approach works.  First, besides the GloVe common crawl embeddings used here, there are several other embedding sets (including the other GloVe embeddings released along with the ones used here, and the Google News word2vec embeddings) that should be considered.  Also, the paper never considers concatenations of large pre-trained embedding sets with each other and/or with the new domain corpus -- such concatenations often give a big boost to accuracy, see :
"Think Globally, Embed Locally—Locally Linear Meta-embedding of Words", Bollegala et al., 2017
https://arxiv.org/pdf/1709.06671.pdf
That paper is not peer reviewed to my knowledge so it is not necessary to compare against the new methods introduced there, but their baselines of concatenation of pre-trained embedding sets should be compared against in the submission.
Beyond trying other embeddings, the paper should also compare against simpler combination approaches, including simpler variants of its own approach.  What if we just selected the one past domain that was most similar to the new domain, by some measure?  And how does the performance of the technique depend on the setting of m?  Investigating some of these questions would help us understand how well the approach works and in which settings.
Minor:
Second paragraph, GloVec should be GloVe
"given many domains with uncertain noise for the new domain" -- not clear what "uncertain noise" means, perhaps "uncertain relevance" would be more clear
The text refers to a Figure 3 which does not exist, probably means Figure 2.  I didn't understand the need for both figures, Figure 1 is almost contained within Figure 2
When m is introduced, it would help to say that m < n and justify why dividing the n domains into two chunks (of m and n-m domains) is necessary.
"from the first m domain corpus" -> "from the first m domains"?
"may not helpful" -> "may not be helpful"
"vocabularie" -> "vocabulary"
"system first retrieval" -> "system first retrieves"
COMMENTS ON REVISIONS: I appreciate the authors including the new experiments against concatenation baselines.  The concatenation does fairly comparably to LL in Tables 3&4.  LL wins by a bit more in Table 2.  Given these somewhat close/inconsistent wins, it would help the paper to include an explanation of why and under what conditions the LL approach will outperform concatenation.