Being able to automatically discover synonymous entities from a large free-text corpus has transformative effects on structured knowledge discovery. Existing works either require structured annotations, or fail to incorporate context information effectively, which lower the efficiency of information usage. In this paper, we propose a framework for synonym discovery from free-text corpus without structured annotation. As one of the key components in synonym discovery, we introduce a novel neural network model SYNONYMNET to determine whether or not two given entities are synonym with each other. Instead of using entities features, SYNONYMNET makes use of multiple pieces of contexts in which the entity is mentioned, and compares the context-level similarity via a bilateral matching schema to determine synonymity. Experimental results demonstrate that the proposed model achieves state-of-the-art results on both generic and domain-specific synonym datasets: Wiki+Freebase, PubMed+UMLS and MedBook+MKG, with up to 4.16% improvement in terms of Area Under the Curve (AUC) and 3.19% in terms of Mean Average Precision (MAP) compare to the best baseline method.
This paper studies the problem of identifying (discovering) synonymous entities. The paper proposes using the "contexts" of the entities as they occur in associated text corpora (e.g. Wiki) in the proposed neural-network based embedding approach for this task. The key novelties of the approach lie in the "matching" system used, where contexts of one entity are matched with that for the other entity to see how well they align with each other (which effectively determines the similarity of the two entities). Experiments are conducted on three different datasets to show the efficacy of the proposed approach.
Overall I found the paper to be an interesting read with some nice ideas mixed in. However I also had some concerns which are highlighted later down below, which I believe if addressed would lead to a very strong work.
Quality: Above average
In general the method seems to work somewhat better than the baselines and the method does have a couple of interesting ideas.
Clarity: Average
I found a few key details to be missing and also felt the paper could have been better written.
Originality: Average
The matching approach and use of the leaky units was interesting tidbits. Outside of that the work is largely about the application of such Siamese RNNs based networks to this specific problem. (The use of context of entities has already been looked at in previous works albeit in a slightly more limited manner)
Significance: Slightly below average
I am not entirely sold on the use of this approach for this problem given its complexity and unclear empirical gains vs more sophisticated baselines. The matching aspect may have some use in other problems but nothing immediately jumps out as an obvious application.
----
Strengths / Things I liked about the paper:
- In general the method is fairly intuitive and simple to follow which I liked.
- The matching approach was an interesting touch.
- Similarly for the "leaky" unit.
- Experiments conducted on multiple datasets.
- The results indicate improvements over the baselines considered on all the three datasets.
Weaknesses / Things that concerned me:
-  (W1) Slightly unfair baselines? One of the first things that struck me in the experimental results was how competitive word2vec by itself was across all three datasets. This made me wonder what would happen if we were to use a more powerful embedding approach say FastText, Elmo, Cove or the recently proposed BERT? (The proposed method itself uses bidirectional LSTMs)
Furthermore all of them are equally capable of capturing the contexts as well. An even more competitive (and fair) set of baselines could have taken the contexts as well and use their embeddings as well. Currently the word2vec baseline is only using the embedding of the entity (text), whereas the proposed approach is also provided the different contexts at inference time. The paper says using the semantic structure and the diverse contexts are weaknesses of approaches using the contexts, but I don't see any method that uses the context in an embedding manner -- say the Cove context vectors. If the claim is that they won't add any additional value above what is already captured by the entity it would be good to empirically demonstrate this.
- (W2) Significance testing: On the topic of experimentation, I was concerned that significance testing / error estimates weren't provided for the main emprical results. The performance gaps seem to be quite small and to me it is unclear how significant these gaps are. Given how important significance testing is as an empirical practice this seems like a notable oversight which I would urge the authors to address.
- (W3) Missing key details: There were some key aspects of the work that I thought were not detailed. Chief among these was the selection of the contexts for the entities. How was this? How were the 20 contexts identified? Some of these entities are likely far more common than just 20 sentences and hence I wonder how these were selected?
Another key aspect I did not see addressed: How were the entities identified in the text (to be able to find the contexts for them)? The paper claims that they would like to learn from minimal human annotations but I don't understand how these entity annotations in the text were obtained. This again seems like a notable oversight.
- (W4) Concerns about the method: I had two major concerns about the method: 
(a) Complexity of method :  I don't see an analysis of the computational cost of the proposed method (which scales quadratically with P the number of contexts); 
(b) Effect of redundant "informative" contexts: Imagine you have a number of highly informative contexts for an entity but they are all very similar to each other. Due to the way the matching scores are aggregated, these scores are made to sum to 1 and hence no individual score would be very high. Given that this is the final coefficient for the associated context, this seems like a significant issue right?
Unless the contexts are selected to be maximally diverse, it seems like this can essentially end up hurting an entity which occurs in similar contexts repeatedly. I would like to see have seen the rationale for this better explained.
(c) A smaller concern was understanding the reasoning behind the different loss functions in the siamese loss function with a different loss for the positive and the negative, one using a margin and one which doesn't. One which scales to 1/4, the other scaling to (1-m)^2. This seems pretty arbitrary and I'd like to understand this.
-(W5) Eval setting : My last concern was with the overall evaluation setup. Knowledge bases like Freebase are optimized for precision rather than recall, which is why "discovery" of new relations is important. However if you treat all missing relationships as negative examples then how exactly are you measuring the true ability of a method? Thus overall I'm pretty skeptical about all the given numbers simply because we know the KBs are incomplete, but are penalizing methods that may potentially discover relations not in the KB.