How to use BERT for finding similar sentences or similar news? 
I have used BERT NextSentencePredictor to find similar sentences or similar news, However, It's super slow. Even on Tesla V100 which is the fastest GPU till now. It takes around 10secs for a query title with around 3,000 articles. Is there a way to use BERT better for finding similar sentences or similar news given a corpus of news articles?
Hi,
BERT out-of-the-box is not the best option for this task, as the run-time in your setup scales with the number of sentences in your corpus. I.e., if you have 10,000 sentences/articles in your corpus, you need to classify 10k pairs with BERT, which is rather slow.

A better option is to generate sentence embeddings: Every sentence / article is mapped to a fixed sized vector. You need to map your 3k articles only once to a vector.

A new query is then also mapped to a vector. In this setup, you only need to run BERT for one sentence (at inference), independent how large your corpus is.

Then, you can use cosine-similiarity, or manhatten / euclidean distance to find sentence embeddings that are closest = that are the most similar.

I released today a framework which uses pytorch-transformers for exactly that purpose:
https://github.com/UKPLab/sentence-transformers

I also uploaded an example for semantic search, where each sentence in a corpus is mapped to a vector and than cosine-similarity is used to find the most similar sentences / vectors:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py

Let me know if you have further questions.<|||||>I think you can use [faiss](https://github.com/facebookresearch/faiss) for storing and finding similar embeddings. <|||||>@nreimers Amazing!! Thank you so much. What you created is a real-life savior! Can this be used for finding similar news(given title and abstract)?  I ran the code and I have the following doubts. 
Which model should I use? 
 bert-large-nli-stsb-mean-tokens vs bert-base-nli-mean-tokens vs bert-large-nli-mean-tokens (what are the datasets on which all these models are trained on?) 

Can I use [faiss](https://github.com/facebookresearch/faiss) to compute the search/distance of the vectors instead of L2/Manhattan/Cosine distances? 

Many thanks to @stefan-it for introducing me to [faiss](https://github.com/facebookresearch/faiss).


<|||||>@nreimers I don't think scipy.spatial.distance.cdist is good enough, it takes a lot of time to compute the results, almost 10 minutes on a corpus of 3.9k news articles. I think I should try using [faiss](https://github.com/facebookresearch/faiss). I don't know anything about [faiss](https://github.com/facebookresearch/faiss) but I will try.<|||||>Hi @Raghavendra15,
regarding the model I sadly cannot be helpful, you would need to test them. In general, sentence embeddings methods (like Inference, Universal Sentence Encoder or my git) work well for short text, i.e., for sentences. For longer text with multiple sentences their performance often decrease and average word embeddings or tf-idf is in many case a much better choice. For longer texts, all these sentence embeddings methods are not really needed.

It would be great if you have some training data. Then, it would be quite easy to fine-tune a model specifically for your task. It should achieve much better performances than the pre-trained models.

I think the issue is not  scipy.spatial.distance.cdist. On a corpus with 100k embeddings and 1024 embedding size, it requires about 0.2 seconds per query (if you can batch queries, even less time is needed).

I think the issue might be the generation of the 4k sentence embeddings? Transformer networks like BERT are extremely slow on CPUs. However, on a GPU, the implementation can process about 2000 sentences per seconds. On a GPU, only about 40 sentences.

But the corpus must only be processed once and can then be stored & loaded from disk. At inference, you just need to generate one embedding for the respective query.

You can of course combine this with faiss. Faiss generates index structures that allow a quick search in vector space and is especially suitable if you have a high number (millions) of vectors. For 4k vectors, scipy takes about 0.008 seconds per queries to find the most similar vectors. 

So either something is really strange with scipy on your computer, or the long run-time comes from the generation of the embeddings.<|||||>@nreimers  Thank you very much for your response. You're absolutely right, most of the time taken is for generating the embedding for 4k sentences. I'm now confused between choosing this model over [XLNet](https://github.com/zihangdai/xlnet), XLNet has achieved the state of the art results.

By your comments on faiss, As long as I have a smaller dataset, results from faiss and scipy won't make any difference? However, If I had millions or billions of news articles then using faiss makes sense right? For smaller datasets, there is no difference in terms of quality of matches between faiss and scipy(the results are the same for computing the distances)? 

I have one important question, If I want to train the model as you suggested which would yield better results, In that case, I should have labeled dataset right? However, for news, I only have titles and abstract about that news. Is there a way to train them without the labels? <|||||>Hi,
XLNet achieved state-of-the-art performance for supervised tasks like classification. But it is unclear if it generates also good embeddings for unsupervised tasks.

In the framework you can choose XLNet, but I was only able to produce results that are slightly below those of BERT.

Others also have problems getting a good performance with Xlnet for supervised tasks, as it appears that it is extremely sensitive to the hyper parameters.

If you have millions of docs, faiss makes sense. With scipy, you get exact scores. With faiss, the scores are fuzzy and the returned most similar vectors must not necessarily be the actual most similar vectors. There can be small variations. But I think the difference will be small.

Often you have in your data some structure, like categories or links between news articles. This structure can be used to fine-tune a model. Let's say you have links linking to similar events. Than you train the network with triplet loss with the two linked articles and one random other article as negative example.

This will give you a vector space where (possibly) linked articles are close. <|||||>@nreimers Thank you very much for your quick response. 
Are the existing model "bert-large-nli-stsb-mean-tokens" better than the google news word2vec [google_news_300](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing), they claim that-" We are publishing pre-trained vectors trained on part of Google News dataset (about 100 billion words). The model contains 300-dimensional vectors for 3 million words and phrases."
Is the pretrained "bert-large-nli-stsb-mean-tokens" better than google's pre-trained news vectors?

For training the existing model to improve results for news similarity, the problem I have is I can't create a dataset to compute triplet loss. For triplet loss to work in the case of news similarity for query news ['**a**'], I need to find a news article ['**b**'] which is similar as a positive example and a dissimilar news article ['**c**'] as a negative example. Like <a,b> positive example and <a,c> negative example.

However, If I run the news every day then, new entities/topics are going to pop up every single day? I need to update my embeddings right? I don't know how to handle this situation. 
<|||||>Google News vectors are just word vectors, you still need a strategy to derive sentence embeddings from these. But as mentioned earlier, average word embeddings is a promising idea for your task. Note, average word embeddings models will be added soon to the repository.

Constantly updating of the model is not needed. News are changing, but the used words remain the same. So training once should give you a model that can be used for a long time. <|||||>@nreimers Thank you very much! Any tentative date by when the average word embeddings will be added to the repository? 

I want to know how to evaluate the results of similar sentences numerically, for example when I use your model to evaluate for a given news, finding similar news in the corpus. 

Is there a way to measure numerically how good the similar sentences are in the below example? I used BLEU score, but the problem is, it's not an accurate measure of similarity. BLEU score doesn't consider the context of the sentence, it just blindly counts whether a word in the query sentence is present in the similar sentence regardless of where the word is placed.

For an item, I get related items.
In the below example, the first title in relatedItems is similar, however, the second item in "relatedItems" is not at all similar which talks about Stephen Colbert and Joe Biden. 
Suppose I use word2vec model for the above task it might give me two totally different sentences as relatedItems, In that case, how can I evaluate both the models and claim numerically which one is better?

Example:

{"title": "Google Is Rolling Out A New Version Of Android Auto - Here's What You Can Expect",
"abstract": "The new Android Auto. Google If you use Android Auto, you're about to receive to a nice upgrade.",
}
"relatedItems": 
[{

 "title": "New Android ransomware is spreading through text messages",
 "abstract": "There\u2019s a new type of Android ransomware making the rounds that leverages SMS to spread, according to a new report from cyberappsecurity com",
},
{
          "title": "Stephen Colbert Brings Curtain Down On Democratic Debates With Joe Biden Tweaks",
          "abstract": "Stephen Colbert closed his second of two live Late Show monologues with a spree of zingers directed at Joe Biden, mixing in plenty for the o",

}
]}



<|||||>Bleu wouldn't be a good measure, because the best similarity metric to find similar news would be: Bleu (of course). 

What you would need is an annotated Corpus. For a given article, get for example the 20 articles with the highest tf idf similarity. Then annotate every pair as similar or not.

With this data you can compare different methods with Ndcg about how well they rank the 20 candidate articles. 

Avg. Word embeddings should be included within the next two weeks to the repo. <|||||>@nreimers When you say -"Bleu wouldn't be a good measure, because the best similarity metric to find similar news would be: Bleu (of course)." 
Do you mean when I get similar news like in the above example, BLEU is the best metric to measure how similar the two news articles are? Please correct me if I understood this wrong.

In the STS benchmark, I saw a pair in the training dataset with gold-standard human evaluated scores. The following paid had a score of 5, however, when I use BLEU scores for 1gram they don't get a score of 1. Instead, they get the following scores. BLEU looks for the exact word to be present in the reference sentence that's the problem I feel. There's no notion of similarity.

s=word_tokenize("The polar bear is sliding on the snow")
reference = [s]
candidate =word_tokenize("The polar bear is sliding across the snow") 
print('Individual 1-gram: %f' % sentence_bleu(reference, candidate, weights=(1, 0, 0, 0)))
print('Individual 2-gram: %f' % sentence_bleu(reference, candidate, weights=(0, 1, 0, 0)))
print('Individual 3-gram: %f' % sentence_bleu(reference, candidate, weights=(0, 0, 1, 0)))
print('Individual 4-gram: %f' % sentence_bleu(reference, candidate, weights=(0, 0, 0, 1)))

Individual 1-gram: 0.875000
Individual 2-gram: 0.714286
Individual 3-gram: 0.500000
Individual 4-gram: 0.400000

reference sentence has 8 words out of which the candidate matches exactly 7 words, so 7/8 score for 1-gram matches.

I'm not sure how the STS benchmarks are evaluated, I'm currently looking into them. If you have any leads or a document I would be more than happy to read them. 

Thank you very much for your help :)

<|||||>No, BLEU is a terrible idea for evaluation.

STS is usually evaluated using Pearson correlation between gold and predicted labels. But Pearson correlation is also a bad idea:
https://aclweb.org/anthology/C16-1009

I strongly recommend to use Spearman correlation for comparison. <|||||>@nreimers Kudos on the COLING paper! It's very well written. In the paper, you have mentioned How Pearson correlation can be misleading or ill-suited for the semantic text-similarity task. However, you did not suggest to use Spearman correlation instead of Pearson correlation. But for me, you suggested me to use Spearman correlation why? (That's my current understanding of the paper)

Can I use the Spearman rank correlation from scipy?
Basically, I want to compare the BERT output sentences from your model and output from word2vec to see which one gives better output.
So there is a reference sentence and I get a bunch of similar sentences as I mentioned in the previous example [ please refer to the JSON output in the previous comments].

Will the below code is the right way to do the comparison? 
In your sentence transformer, you have used the same below package in SentenceEvaluator class. I couldn't figure out how to use that class for my comparison. 

Will you please give me some idea in this regard?

Example code:
from scipy.stats import spearmanr
x = [1, 2, 3] ---> I will use BERT and word2vec embeddings here.
x_corr = [2, 4, 6]
corr, p_value = spearmanr(x, x_corr)
print (corr)
<|||||>Hi @Raghavendra15 
The issue with pearson correlation is, that it assumes a linear correlation between the system output and gold labels. Adding a montone function to the system output can change the scores (make them better or worse), which does not really make sense in applications.

Assume you have a systems that predicts the perfect gold scores, however, the output is output=sqrt(gold_label).

This system would get a really low Pearson correlation. However, for every application, this system would be perfect, as it predicts the gold labels. With Spearman correlation, you don't have this issue. There, just the ranking of the scores are important.

In general I think the STS tasks (or the STS benchmark) are not really well suited to evaluated approaches. The STS tasks with Pearson/Spearman correlation weights every score similar, but in applications, we are often only interested in certain examples.

For example, if we search for pairs with the highest similarity, then we don't care how the scores are for low similarity pairs. A system that gives a perfect score for high similarity pairs and a random score for low similarity pairs would be great for this application. However, this system would get a low Pearson/Spearman correlation, as it fails to correctly order the somewhat-similar and unsimilar pairs.

If you want so estimate the similarity of two vectors, you should use cosine-similarity or Manhatten/Euclidean distance. 

Spearman correlation is only used for the comparison to gold scores.

Assume you have the pairs:
x_1, y_1
x_2, y_2

...
for every (x_i, y_i) you have a score s_i from 0 ... 1 indicating a gold label score for their similarity. 

You can check how good the embeddings are by computing the cosine similarity between the embeddings for (x_i, y_i) and then you compute the Spearman correlation between these computes cosine similarity scores and the gold score s_i.

Note: Currently I add methods to compute average word embeddings and similar methods to the repository. So a comparison will become easier.<|||||>@nreimers  Last week you added the methods to compute average word embeddings should I use that method when I get a sentence embedding or will there be a pre-trained average word embedding weights? 
In the below code I will get the embeddings once I pass the input strings. Should I use the compute avg word embedding method on top of this?

corpus = ['A man is eating a food.',
          'A man is eating a piece of bread.' ]
corpus_embeddings = embedder.encode(corpus)
or 
By any chance, pre-trained avg-word embedding weights will be uploaded to the repository by any time this week.  <|||||>Hi @Raghavendra15 
I just uploaded v0.2.0 to github and PyPi:
https://github.com/UKPLab/sentence-transformers

You can update with pip install -U sentence-transformers

I added an example for average  word embeddings (+a DAN layer that is trainable):
https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_stsbenchmark_avg_word_embeddings.py

You can also use it without the DAN layer. There is also a tokenizer implemented that allows the usage of the word2vec Google News vectors. These vectors contain phrases  like 'New_York'. These phrases are detected by the tokenizer and mapped to the correct embedding for New_York. But there is currently no example for this in the repo. If you need help, let me know.

To get avg. word embeddings only (without DAN), the code must look like this:
```
# Map tokens to traditional word embeddings like GloVe
word_embedding_model = models.WordEmbeddings.from_text_file('glove.6B.300d.txt.gz')

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

corpus_embeddings = model.encode(corpus)
```

Next release will update include support for RoBERTa and add other sentence embeddings methods (like USE, LASER), which will be trainable.<|||||>@nreimers Thank you very much! You spoke my mind with RoBERTa, I was about to ask you about it. But with the avg-embedding approach, I won't be using BERT at all right? 

In addition to that, I won't be training the model. I don't think I fully understand this. Earlier I would pass a pretrained weight model into SentenceTransformer, however now I won't pass anything related to BERT, does that mean I won't be using BERT?<|||||>@Raghavendra15 
The framework offers you a lot of flexibility. You can choose between the following embedding approaches:
- BERT or XLNet (RoBERTa and other will follow)
- Traditional word embeddings like GloVe, word2vec etc.

Then, you can choose between different pooling modes: Mean pooling, max pooling, usage of the CLS token for BERT / XLNet.

Finally, if you like, you can add feed-forward networks to create a deep-averaging network.

If you have training data, I can recommend this combination:
BERT + mean-pooling

This gave the best performance for many cases. 

If you have training data, you need a low computation time and performance is not that important, choose this combination:
GloVe embeddings (or something similar) + mean-pooling + 1 or 2 dense layers

If you don't have training data, choose:
GloVe embeddings (or something similar) + mean-pooling 

As you can see, there are various options you can choose from, depending if you have training data and how important is a high speed vs. a good performance.

Once I have RoBERTa integrated, how suitable it is for the generation of sentence embeddings. My experiences with XLNet was that the performance is slightly below the performance of BERT for sentence embeddings. Maybe RoBERTa is better for sentence embeddings, maybe not.

Averaging BERT without fine-tuning on data gave really poor results. However, what you can of course try, is to use one of the existent pretrained BERT models like 'bert-base-nli-mean-tokens', which is BERT+mean-pooling, fine-tuned on NLI data to generate meaningful sentence embeddings.
<|||||>@nreimers Thank you very much! Why didn't you choose (word2vec) Google news vectors? Is there any particular reason for choosing Glove embedding over word2vec? I'm curious to know how RoBERTa will perform! 😃<|||||>@Raghavendra15 
There are two reasons:
1) Google news word2vec is quite large, it requires about 12 GB of RAM to read it in. Not that ideal for an example script. GloVe embeddings are about 10 times smaller.
2) In most of my experiments, the Google news word2vec vectors did not yield good performances. GloVe embeddings were often a bit better. I especially like the embeddings by Levy et al (trained on dependencies) and by Komninos. I also conducted a larger comparison between word embeddings (https://arxiv.org/abs/1707.06799, Table 5).

But note, using the Google news word2vec vectors is quite easy. In training_stsbenchmark_avg_word_embeddings.py replace
```
word_embedding_model = models.WordEmbeddings.from_text_file('glove.6B.300d.txt.gz')
```
with
```
word_embedding_model = models.WordEmbeddings.from_text_file('GoogleNews-vectors-negative300.txt.gz')
```

First experiments with RoBERTa are done: On STSbenchmark, it increases the Spearman correlation by about 1 - 2 percentage points. I will see how it will perform on other datasets.

Best, Nils Reimers<|||||>This issue is very interesting, thanks for sharing your experiments and framework @nreimers!<|||||>@nreimers I read your paper on word embedding comparison, however, when I saw the GLEU scoreboard for STS benchmark Glove scored very less compared to word2vec, Isn't it contradictory to your paper? Also in your paper, the comparisons are on a certain set of tasks like Entity Recognition, NER but not on Semantic Textual Similarity. I don't know much about it, I'm trying to learn. Do my questions make sense?
 
Is there any significant difference between using glove.840B.300d.zip (contains 840 billion words vectors trained on the common crawl ) vs glove.6B.300d.txt.gz (contains 6 billion words vectors wikipedia+Gigaword), Is it like more words the better? also, they're trained on different datasets, will that make a huge difference when applied to news similarity?<|||||>See the GloVe website / paper for the differences. 6B was trained on 6 billion words from Wikipedia, 840B was trained on 840 Billion words from common crawl.

It depends on the task and data which one is more suitable. If you have a lot of rare words, and those play an important role for your task, 840B is often better. If you have clean data / only common words are important for your task, 6B often works better. 

However, the differences are often only minor between the two versions. 

In my paper I only compare embeddings for supervised task, only for sequence tagging. 

In unsupervised tasks, you can get completely different results. Further, how word embeddings are averaged has a big impact. Some authors don't ignore stop words, instead they propose some complicated weighting scheme. If stop words are ignored, performances can be improved up tp 10 percentage points, sometimes outperforming complex weighting approaches. 

Best, 
Nils Reimers <|||||>Thank you for your work, Nils, it is brillant! 

I would like to design a sentence level semantic search engine using email data (Enron dataset). 

I am still a little bit confused about how I should be fine-tuning models on such dataset (maybe I am missing something obvious).

Thanks.

Gogan
<|||||>@ggndtes In general BM25 will be really hard to beat on this type of task. See this paper where they compare sentence embeddings with BM25 on an end-to-end retrieval task (given: question, find similar / duplicate questions in a large corpus):
https://arxiv.org/pdf/1811.08008.pdf

A complex sentence embedding method only achieves 1 - 2 percentage points improvement against BM25 (Table 2, Dual Encoder Paralex vs. Okapi BM 25).

Especially  if you have more than just a sentence, carefully constructed BM25 for example with Elasticsearch is really really hard to beat. If you are interested in a production system, I would highly recommend to first try Elasticsearch (or similar), beating it will be difficult.

Back to your question how you can tune it:
The big question narrows down to: What are your queries, what are your documents. Are your documents complete emails? Or only email subjects? Or only sentences within emails?

Are your queries inputs from the user, email subjects or complete emails?

In general you would need to construct same sort of similarity. Currently I can only think of imperfect method to create similarity labels. One option would be: Triplet loss with 2 emails from the same inbox vs. one random other subject. But this would I think create rather bad embeddings.

Currently I can't think of a good method to create similarity labels for that dataset. And as mention, even with perfect labels, it will be really hard to beat BM25. 

Best,
-Nils Reimers
  


<|||||>@nreimers The sentence encoder actually takes quite a lot of time to load the Glove embeddings, Is there a way where I can make it load from the disk or make it faster?<|||||>@Raghavendra15 When you run the code the first time, the embeddings are downloaded and stored in the path of the script. In follow-up executions, the embeddings file is loaded from disk.

GloVe embeddings are quite large, so loading it can take some time.

There are two ways to speed it up:
1) Limit the vocab size, i.e., don't load all the ~400k embeddings. Pass the parameter 'max_vocab_size' to the method 'from_text_file' when called.
2) Save the WordEmbeddings model to disc. In follow-up executions, you can load the (binary) model directly from disc and you don't have to read in and parse in the text file.

Should work something like this:
```
word_model = WordEmbeddings.from_text_file('my-glove-file.txt')
word_model.save('my/output/folder/GloveWordModel')

# In follow-up calls, should be faster
word_model = WordEmbeddings.load('my/output/folder/GloveWordModel')
```<|||||>@nreimers Wow!! It works blazingly fast! 
I was trying to play with the below code. Thank you very much for the help :) 
Code in In WordEmbeddings.py file:
```
 with gzip.open(embeddings_file_path, "rt", encoding="utf8") if embeddings_file_path.endswith('.gz') else open(embeddings_file_path, encoding="utf8") as fIn:
            iterator = tqdm(fIn, desc="Load Word Embeddings", unit="Embeddings")
            for line in iterator:

```
Also, can I load the model similar to that for BERT pre-trained weights? such as the below code?

`embedder = SentenceTransformer('bert-large-nli-stsb-mean-tokens')`
Can I load the above pre-trained weights somehow just like you have `load` method for glove weights?

Is the avg embedding with Glove better than "bert-large-nli-stsb-mean-tokens" the BERT pre-trained model you have loaded in the repository? How's RoBERTa doing? Your work is amazing! Thank you so much again! 
<|||||>@Raghavendra15 Sure you can:
```
word_embedding_model = models.WordEmbeddings.from_text_file('glove.6B.300d.txt.gz')

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)


model = SentenceTransformer(modules=[word_embedding_model, pooling_model])
model.save('my/output/folder/avg-glove-embeddings')

# Load the Model:
model = SentenceTransformer('my/output/folder/avg-glove-embeddings')
```

Which model is better depends extremely on your data and on your task. The BERT models work good if you have clean data, which is not too domain specific and rather descriptive. This is due to the nature on which data it was fine-tuned (on NLI dataset).

Average GloVe embeddings works I think better if you have noisy data, really domain specific data or very short sentences or very large paragraphs. 

Experiments with RoBERTa are finished. Paper will be uploaded next week to arxiv. In my experiments, I could not observe a major difference between BERT and RoBERTa for sentence embeddings: Sometimes BERT is a little bit better, sometimes RoBERTa. But nothing that is significant. XLNet was so far in general worse than BERT.

Best
-Nils Reimers<|||||>@nreimers Thanks! but my question is how can make the pretrained BERT model faster like loading the below model.
`embedder = SentenceTransformer('bert-large-nli-stsb-mean-tokens')
`
When I run the encoder for BERT it takes a lot of time like 10-15 minutes for 4k sentences. 
`embedder.encode(corpus)` ---> This takes around 10minutes for "bert-large-nli-stsb-mean-tokens"
However, Glove model does the job in 30 secs. bert-large-nli-stsb-mean-tokens is similar to glove pretrained word vectors right? Then is there a way to convert speed up the BERT sentence encoder?<|||||>@Raghavendra15 No, the BERT model and average GloVe embeddings are completely different.

GloVe embeddings have one vector for each word in a language, for example, the word 'apple' is mapped to the vector 0.31 0.42 0.15 ....

To compute avg. GloVe embeddings, you just perform some memory lookup: Every word is mapped to the vector and then you compute the mean values.

BERT (https://arxiv.org/abs/1810.04805) is much more complex: Words in a sentence are first broken down to subwords, which are than mapped to vectors (which is the fast part).

But after that, a transformer network is run over the complete sentence: For BERT-base it has 12 layers, for BERT-large, it has 24 layers. This produces vectors for each word which depend on the context of the complete sentence.

If you have the two sentences:
- Apple is a healthy fruit
- Apple presented their new iPhone

With GloVe, Apple are mapped in both cases to the same vector. With BERT, the two Apple words are mapped to different embeddings. In the first case, it is mapped closer to words like Banana, Mango etc., in the second sentence, it is mapped closer to words like Microsoft, Google etc.

But this comes with a cost: Transformer networks are rather slow. This is especially true if you have only a CPU or an older GPU.

On a CPU, you can process with BERT about 80 sentences / second (with GloVe, more than 5k). On a Nividia V100 GPU, the speed is a bit better: About 2000 sentences / second (BERT-base).

The runtime for transformer networks is quadratic with the sentence length. If your sentence is twice as long, the runtime increases 4x.

So the only ways to speed-up the BERT model:
- Try to figure out what the optimal batch size is for your system (you can pass the batch size as a parameter to encode())
- Use the base, not the large model. The large model is multiple times slower than the base model.
- Be careful with your sentences lengths. Maybe truncate your sentences
- Get a better / faster GPU (or multiple GPUs). Running BERT on CPU is horrible.

I hope this of some help for you.

Best regards
-Nils Reimers

<|||||>This is an outstanding explanation Nils – you should blog or tweet, I'm sure lots of people would be interested in reading more from you!<|||||>@nreimers Brilliant explanation! :D You're a life saviour :-)
I need your help with this issue. Can I use sentence transformer for this case?
https://github.com/huggingface/pytorch-transformers/issues/1170<|||||>@nreimers Very patient brilliant explanation. Wish u a happy life.<|||||>@nreimers 
Let's say you have a sufficient training set for information retrieval, such as that from fever.ai.

We used black-box Bayesian Optimization to train BM25 on Elasticsearch... producing close to the results described in the SOTA evidence retrieval from the UKP-Athene team, but were still a few % off SOTA, without entity extraction or any other ML preprocessing. 

Shouldn't it be the case that a well trained encoder transformer with cosine-loss, with specific weights for a query and document / sentence in the result set, should be able to beat an arbitrary algorithm like BM25?

And that it could be deployed at scale using faiss or hsnw?<|||||>Hi @pertschuk 
If the recall of BM25 is quite good, I would aim for re-ranking instead of a full semantic search.

In re-ranking, you retrieve e.g. 100 documents with your BM25 algorithm. Then, you run BERT to compare each document with your query to get one score (0...1). 
Next, you sort these scores.

Your original ranking from BM25 is then replaced with these BERT-based scores ranking.

Sentence embeddings often have challenges in information retrieval as the false positive probability is higher than BM25. I.e., if you compare two dissimilar sentences with sentence embeddings, the probability of getting a high similarity score is higher for approaches like Sentence-BERT / InferSent / USE, than it is for BM25.

In Information Retrieval, you usually have a large set of unrelated docs, i.e., this higher false positive rate leads to really bad consequences that you find many unrelated documents, leading to a performance usually lower than BM25.

The re-ranking approach prevents this to happen: BM25 gives you a rather clean candidate set, and your neural re-ranking approach can then do the hard work and determine which of the n documents matches the query the  best.

Best regards
Nils Reimers<|||||>Great thank you, this makes sense. 

We are currently using reranking on top 9 documents, but maybe could increase this number since our reranking recall is quite high ~.95 based on a  RoBERTA regression model. 

link: https://github.com/koursaros-ai/koursaros/blob/master/examples/pipelines/factchecking/services/scorer/__main__.py

I guess then the challenge becomes the scale of re-ranking, because there would be ~700 sentences to rerank with this larger set, and we can maybe run 100/s on SoTA transformer.

I wrote a FEVER dataset loader and am currently training a sentence reranking model based on your cosine loss, I am hoping to achieve the greater performance afforded by precomputing embeddings and running KNN to rerank, I will publish results here when I have them. <|||||>Yes, larger candidate sets can actually be quite interesting.

What you can also try is the faster, destilled BERT from hugging face. It achieves similar results like BERT, but is faster.

Sometimes, a larger set with worse (but cheaper) models achieves better overall results than a small set with a better (but expensive) model.

Best
-Nils Reimers<|||||>> Hi,
> BERT out-of-the-box is not the best option for this task, as the run-time in your setup scales with the number of sentences in your corpus. I.e., if you have 10,000 sentences/articles in your corpus, you need to classify 10k pairs with BERT, which is rather slow.
> 
> A better option is to generate sentence embeddings: Every sentence / article is mapped to a fixed sized vector. You need to map your 3k articles only once to a vector.
> 
> A new query is then also mapped to a vector. In this setup, you only need to run BERT for one sentence (at inference), independent how large your corpus is.
> 
> Then, you can use cosine-similiarity, or manhatten / euclidean distance to find sentence embeddings that are closest = that are the most similar.
> 
> I released today a framework which uses pytorch-transformers for exactly that purpose:
> https://github.com/UKPLab/sentence-transformers
> 
> I also uploaded an example for semantic search, where each sentence in a corpus is mapped to a vector and than cosine-similarity is used to find the most similar sentences / vectors:
> https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py
> 
> Let me know if you have further questions.

Hi, Can t

> Hi,
> BERT out-of-the-box is not the best option for this task, as the run-time in your setup scales with the number of sentences in your corpus. I.e., if you have 10,000 sentences/articles in your corpus, you need to classify 10k pairs with BERT, which is rather slow.
> 
> A better option is to generate sentence embeddings: Every sentence / article is mapped to a fixed sized vector. You need to map your 3k articles only once to a vector.
> 
> A new query is then also mapped to a vector. In this setup, you only need to run BERT for one sentence (at inference), independent how large your corpus is.
> 
> Then, you can use cosine-similiarity, or manhatten / euclidean distance to find sentence embeddings that are closest = that are the most similar.
> 
> I released today a framework which uses pytorch-transformers for exactly that purpose:
> https://github.com/UKPLab/sentence-transformers
> 
> I also uploaded an example for semantic search, where each sentence in a corpus is mapped to a vector and than cosine-similarity is used to find the most similar sentences / vectors:
> https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py
> 
> Let me know if you have further questions.

Can this use GPUs, if so how ?<|||||>Hi @duttsh,
Yes, GPU is supported out of the box. You just need the necessary Cuda drivers and then you can train / perform inference on the GPU without any changes 

Best regards 
Nils Reimers <|||||>Thanks Nils

Sent from my iPhone

> On Oct 6, 2019, at 1:43 AM, Nils Reimers <notifications@github.com> wrote:
> 
> Hi @duttsh,
> Yes, GPU is supported out of the box. You just need the necessary Cuda drivers and then you can train / perform inference on the GPU without any changes
> 
> Best regards
> Nils Reimers
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
<|||||>@nreimers (Nils) one more question, when will you have pre-trained models of RoBERTa available ? or if they are please send me the name.<|||||>@duttsh I can try to upload it, but in my experimemts I didn't see any improvements from Roberta for sentence embeddings.

Best regards 
Nils Reimers <|||||>Thanks, can you please upload. Also I believe Roberta will increase the accuracy of inference. Right ?

Sent from my iPhone

> On Oct 6, 2019, at 7:18 PM, Nils Reimers <notifications@github.com> wrote:
> 
> @duttsh I can try to upload it, but in my experimemts I didn't see any improvements from Roberta for sentence embeddings.
> 
> Best regards
> Nils Reimers
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
<|||||>@duttsh In my experiment, I didn't observe any differences between BERT and RoBERTa when used for different sentence embeddings tasks. <|||||>@nreimers thanks. If you could share he name if your RoBERTa model, would be great.<|||||>After a couple months of research, the best approach I've found for building semantic search is to integrate with an existing BM25 search platform such as Elasticsearch, and then **rerank** the top n results using a neural network regression trained to score a <query> <passage> combination, on a dataset such as [MS MARCO.](http://www.msmarco.org/)

Per @nreimers comment, something like BM25 produces a cleaner result set, and training a model to look at query passage pairs at the same time rather than training a cosine loss and comparing precomputed vectors enables it to use attention to more accurately rank passages.

Check out this project that implements such a system: https://github.com/koursaros-ai/nboost<|||||>Took me a long time to reply but thanks so much @nreimers for your incredibly clear explanations and responses.

Thanks also to @pertschuk for sharing the results of your research, this is very helpful.<|||||>Hi,
I'm looking into the question of finding prior art for patents.
This means for one patent application (around 20pages) we would like to find the closest 100 patents in a corpus of 100 million patents. The search results of patent offices could be used as training material.
We thought about tf-idf, word2vec, GloVe etc. So far transformers like BERT seemed to be too slow for such a task. 
Now with SBERT and SRoBERTa and powerful AI accelerators, we ask ourselves, if we shouldn't be so quick to exclude transformers.
Any advice? Has anyone applied SBERT to such an amount of data? Anyone using AI accelerators such as Jetson?
<|||||>@wolf-tag 
The two biggest issues with my research into building a transformer cosine loss solution based on SBert at scale (I was working with ~6million wikipedia articles, much smaller than all of the patents).

1. Evaluating the solution. To rebuild all of the 6 million vectors and put them into a FAISS index takes like (200/s to encode) takes like 6-8 hours, and then more time to actually query your test set and calculate something like MRR. Building a good model often requires **dozens** of evals, tweaking, etc.
2. Memory usage. There are various compression methods, but currently vector indexes are pretty memory-hungry. https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors. This has possible solutions. It seems like probably to make this scalable, you would need much smaller embeddings than like 1024 for BERT large.

If you are well funded and have lots of GPU/ TPU and memory, it's feasible, and I would look at Patent-BERT, and incorporate that into sentence transformers. 

One final thought to keep in mind - I have found that almost everything out there, patents included, have summaries (abstracts). At an even more micro scale, humans often tend to summarize a paragraph with the first sentence. You can leverage this to optimize your solutions, by choosing to look at the summary text instead of **all** of it.<|||||>Thank you for your quick reply. 

Did I get it right: you used one vector for each wikipedia article which is the result of SBERT's pooling operation? 
<|||||>Hi @wolf-tag 
Personally I think tf-idf / BM25 is the best strategy for your task, due to various reasons.

**First**, it is important to differentiate between false positive and false negative rates:
False positive: A non-similar pair of docs is judged as similar, even they are not similar.
False negative: A similar pair of docs is judged as dissimilar.

TF-IDF/BM25 has a low false positive rate and a high false negative rate, i.e., if a pair is judged as similar, there is a high chance that they are actually similar.

Sentence embeddings methods (avg. GloVe embeddings, InferSent, USE, SBERT etc.) have a reverse characteristic: high false positive rates, low false negative rates. It seldom misses a similar pair, but a pair judged as similar must not necessarily be similar.

For Information Retrieval, you have an extrem imbalance. You have 1 search query and 100 Mio. documents, i.e., you perform 100 Mio pairwise comparisons.

Sentence embeddings with a high false positive rate will return many pairs where the embeddings think they are similar, but they are not. Your result set of 10 documents will be often completely garbage.

TF-IDF / BM25 might miss some relevant documents, but the 10 document you will find will be of high quality.

**Second**, in my experience, sentence embeddings methods work best for sentences. For (longer) documents, the results are often not that great. Here, word overlap (with tf-idf / BM25) is really hard to beat.

**Third**: In our experiments in Question Answering (given a question, find in Millions of answers on StackOverflow the correct one), TF-IDF / BM25 is extremely hard to beat. It often performs much better than sentence embeddings methods + it is much quicker.

So far our experiments with end-to-end representation learning for information retrieval rather failed.

What works quite good is a re-ranking approach: You use  BM25 to retrieve the top 100 documents. Than, you take a neural approach like BERT to re-rank these 100 results and you present the top-10 results (the 10 results with the highest score according to the neural re-ranker) to the user. This often gives a nice boost to pure BM25 ranking, and the runtime is not too-bad, as you must only re-rank 100 documents.

Best regards
Nils Reimers <|||||>Dear Nils,

thank you for your detailed explanation. 

Indeed, recent publications for the prior art task do hardly show any improvements when using word2vec, GloVe or doc2vec compared to tf-idf.

I was just curious as Google now uses BERT for the search engine, and I suppose they are more interested in high precision than in high recall, so somehow they seem to master the high false positive rates.  Maybe they do so as recommended by you (re-ranking). 

I hoped that one of the newer methods would somehow have a positive impact on this task. Just wishful thinking, I fear.<|||||>Hi @wolf-tag 
an interesting paper could be this:
https://arxiv.org/abs/1811.08008

In Table 2 you see, that BM25 outperforms untrained sentence embeddings methods like avg. word2vec. If you have a lot of training data, you can tune the dual encoder so that it performs better than BM25 for the tested task (finding similar questions).

However, the task of finding similar questions involves rather short documents (often only a sentence). For longer documents, I would guess that BM25 still outperforms sentence embeddings methods.  

In the paper it would have been interesting to compare the methods also against neural re-ranking, to see if the trained end-to-end retrieval is better or worse than the BM25 + re-ranking approach.

Best regards
Nils Reimers

<|||||>Thank you for the hint.

If TF-IDF / BM25 is still the best option for long documents, there seems to be a lot of room for improvement for future research, as this method does not use context, does not follow any semantic approach such as WSD, WordNet or synsets, does neither use trained models nor exploit available training data and does not use any language specific resources (e.g. stemming or noun phrase identification). Maybe some kind of challenge is needed to encourage research in this field.

Best regards,

Wolfgang
<|||||>Hey @nreimers deep thanks for all the info!
(Hopefully) quick question: what would be the optimal setup to find similarities (and build a search engine) between objects defined by a combination of senses?

For instance, consider a DB:
Object 1: "pizza", "street food", "Italian cuisine"
Object 2: "khachapuri", "street food", "Georgian cuisine", "cheese", "bread"

And then, a query "cheesy street food".

I'm using USE + hnswlib now, works pretty good, but only if the query string is more than 1 word. The more words, the better.<|||||>Hi @realsergii 
Not sure if USE is the best match for that task. From the given example, I would again think that you would get quite far with BM25 and for example Elasticsearch. Elasticsearch is quite great for indexing complex objects and search over them.

Of course you would need to tune the search a bit, e.g. that longer n-grams give higher scores, maybe combined with stemming / lemmatization of words.

Otherwise, for individual words, I think word embeddings (like Word2vec / GloVe) are quite great. Sentence embeddings often have difficulties to give a good representation for words or short phrases, as these systems were not trained for it.

Also this repo could be interesting, which combines Elasticsearch BM25 with BERT re-ranking:
https://towardsdatascience.com/elasticsearch-meets-bert-building-search-engine-with-elasticsearch-and-bert-9e74bf5b4cf2

This could potentially also be combined with a simple average word embedding re-ranking approach.

I hope that helps.

Best
Nils Reimers
<|||||>@nreimers thanks Nils!
Just one more clarification - what would change if in my DB I replace each word/phrase with the 1st sentence of Wikipedia entry which is the closest to the respective word/phrase ?
So in that case, would USE or SBERT be a good choice?<|||||>Hi @realsergii 
Sounds a bit complicated and you would have several other issues (how to find the correct article, what about small spelling variances).

Word embeddings are quite strong on finding similar words. As the context is rather small, I don't see too much benefit from using a sentence embedding methods to disambiguate words. 'Cheese' in your context will most often refer to the food, and not to e.g. a company or a strategy in a computer game.

Best
Nils Reimers
<|||||>Thanks @nreimers 
My idea is not just to find similar words/phrases, but to find similar senses.
E.g. "welding" is similar to "joining", "building", in my understanding.
In order to comprehend this, a machine needs to know what are all of those concepts, described in more basic words, right? 
One way to teach a machine is to create a vector from a sentence where the sense is described by Wikipedia (and thus, in more basic concepts).
Other way is to just get a sense (as a vector) (by word/phrase) from a model trained on Wikipedia and other sources.
This is my understanding. 

Please suggest what sounds better.<|||||>Hi @realsergii 
That is exactly what words embeddings are great for: to find similar words, e.g. welding is similar to joining / building.

Mapping words to Wikipedia definitions sounds unnecessary complicated and I doubt you get good results with this (compared to simple word embeddings). At the end, as you have a fixed word to Wikipedia article mapping, you will get a fixed word -> vector mapping. But it is much more complicated and the quality will be much lower.

I would train word2vec / GloVe on large amount of text from your domain and then you can use these word embeddings for comparing word similarities.



<|||||>That’s why using word embedding. Because tf-did just find lexicon
overlapping but not the similar semantic meaning

On Mon, Dec 23, 2019 at 03:58 Nils Reimers <notifications@github.com> wrote:

> Hi @realsergii <https://github.com/realsergii>
> That is exactly what words embeddings are great for: to find similar
> words, e.g. welding is similar to joining / building.
>
> Mapping words to Wikipedia definitions sounds unnecessary complicated and
> I doubt you get good results with this (compared to simple word
> embeddings). At the end, as you have a fixed word to Wikipedia article
> mapping, you will get a fixed word -> vector mapping. But it is much more
> complicated and the quality will be much lower.
>
> I would train word2vec / GloVe on large amount of text from your domain
> and then you can use these word embeddings for comparing word similarities.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/876?email_source=notifications&email_token=AIEAE4GQI2RXKMS4HDVY2Y3QZ7BIBA5CNFSM4IGKGJT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHPYXRA#issuecomment-568298436>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4EKRBKQQRZYQTNZXKDQZ7BIBANCNFSM4IGKGJTQ>
> .
>
<|||||>Thanks! So it feels like Elasticsearch 7.3+ with a bunch of `dense_vector`s from GloVe for all components of my objects (e.g. "khachapuri", "street food", "Georgian cuisine", "cheese", "bread") is the most proper data structure for my system (semantic search engine). 

I don't even need MySQL (for text representation storage) + separate index based on e.g. Faiss (for vectors and index) and don't need to sync them. Everything can be inside Elasticsearch (however the query speed will be slower than Faiss but I can live with that for now).

@nreimers just want to clarify that I don't need to even look into BERT/USE etc direction, right?<|||||>@realsergii 
If your queries and documents are only words or short phrases, I think there is no benefit from using BERT / USE. Sentence embeddings can be helpful when you have more text (at least a sentence) and when words can be used ambiguously (like the word apple).<|||||>@realsergii I mentioned it prior in this thread, but if you're using elastic search as your backend, check out [NBoost](https://github.com/koursaros-ai/nboost), which acts as a proxy on top of ES and uses BERT to rerank the n top results. 

We recently released TinyBERT distilled version of the base models which are about 10x faster (critical when it comes to search). See https://arxiv.org/abs/1909.10351 for distilling custom models by the same method. <|||||>@Raghavendra15  @nreimers  Did you end up trialing it out with Faiss ? What were the results. ?
I have a similar use case where I have a domain dataset (about 100k english sentences) related to fires, I want to find synthetic multilingual sentences in different languages (arabic, italian, chinese etc.).  My thought was to download the Wikipedia corpus (source) for each language and embed both wikipedia and my fire data and find synthetic sentences.  

By following this example [Semantic Similarity](https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py)
- I was able to download the multilingual trained models `distiluse-base-multilingual-cased` 
- Embed about 4.2 Million Arabic sentences (took about 7hrs on p2.xlarge instance, 85% GPU utilization) and 100k Fire Sentences (took couple of minutes)

This is where it hangs/very slow - 
- Running the similarity using cdist seems to run forever, had to cancel after running it for a day. I did not expect for it to take this long. Even though it was very straightforward. Figuring there should be a more optimized way of doing this.  

Is there something wrong with the steps I have taken, appreciate any help. 

Cheers !
Ayub<|||||>Update: I ran it with faiss library using FlatIndex (as it give the most accurate results). On a p2.xlarge instance it was amazingly fast- building and searching took only 30 mins. I could not compare the results to scipy's cdist but for a sample of 10,000. I saw than >90% of the results lie in the top 5 matches found by faiss distance.<|||||>@Raghavendra15 @mohammedayub44  FYI, two relevant papers that recently came out from Google and Microsoft:
- Pre-training embeddings for large-scale retrieval https://arxiv.org/abs/2002.03932
- TwinBERT: distillation for efficient retrieval https://arxiv.org/abs/2002.06275<|||||>How to train BERT on LinkedIn pages? <|||||>> Hi @wolf-tag
> Personally I think tf-idf / BM25 is the best strategy for your task, due to various reasons.
> 
> **First**, it is important to differentiate between false positive and false negative rates:
> False positive: A non-similar pair of docs is judged as similar, even they are not similar.
> False negative: A similar pair of docs is judged as dissimilar.
> 
> TF-IDF/BM25 has a low false positive rate and a high false negative rate, i.e., if a pair is judged as similar, there is a high chance that they are actually similar.
> 
> Sentence embeddings methods (avg. GloVe embeddings, InferSent, USE, SBERT etc.) have a reverse characteristic: high false positive rates, low false negative rates. It seldom misses a similar pair, but a pair judged as similar must not necessarily be similar.
> 
> For Information Retrieval, you have an extrem imbalance. You have 1 search query and 100 Mio. documents, i.e., you perform 100 Mio pairwise comparisons.
> 
> Sentence embeddings with a high false positive rate will return many pairs where the embeddings think they are similar, but they are not. Your result set of 10 documents will be often completely garbage.
> 
> TF-IDF / BM25 might miss some relevant documents, but the 10 document you will find will be of high quality.
> 
> **Second**, in my experience, sentence embeddings methods work best for sentences. For (longer) documents, the results are often not that great. Here, word overlap (with tf-idf / BM25) is really hard to beat.
> 
> **Third**: In our experiments in Question Answering (given a question, find in Millions of answers on StackOverflow the correct one), TF-IDF / BM25 is extremely hard to beat. It often performs much better than sentence embeddings methods + it is much quicker.
> 
> So far our experiments with end-to-end representation learning for information retrieval rather failed.
> 
> What works quite good is a re-ranking approach: You use BM25 to retrieve the top 100 documents. Than, you take a neural approach like BERT to re-rank these 100 results and you present the top-10 results (the 10 results with the highest score according to the neural re-ranker) to the user. This often gives a nice boost to pure BM25 ranking, and the runtime is not too-bad, as you must only re-rank 100 documents.
> 
> Best regards
> Nils Reimers

I am working on a use case where I need to get similar documents (2-3 pages average), when I upload a 1 page document. For me reducing false negatives is a priority, at the same time I don't want too many false positives. Can I first implement an embedding model to get let's say 200 similar documents and then apply TFIDF/BM25 to filter out irrelevant documents <|||||>I just recently started on NLP and "AI" and have been following this thread. Having a similar use case (less than 10k documents --> find similar documents and also do a multi-label classification) I am very interested in your opinion on BERT-AL:

https://openreview.net/pdf?id=SklnVAEFDB<|||||>> @Raghavendra15 @nreimers Did you end up trialing it out with Faiss ? What were the results. ?
> I have a similar use case where I have a domain dataset (about 100k english sentences) related to fires, I want to find synthetic multilingual sentences in different languages (arabic, italian, chinese etc.). My thought was to download the Wikipedia corpus (source) for each language and embed both wikipedia and my fire data and find synthetic sentences.
> 
> By following this example [Semantic Similarity](https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py)
> 
> * I was able to download the multilingual trained models `distiluse-base-multilingual-cased`
> * Embed about 4.2 Million Arabic sentences (took about 7hrs on p2.xlarge instance, 85% GPU utilization) and 100k Fire Sentences (took couple of minutes)
> 
> This is where it hangs/very slow -
> 
> * Running the similarity using cdist seems to run forever, had to cancel after running it for a day. I did not expect for it to take this long. Even though it was very straightforward. Figuring there should be a more optimized way of doing this.
> 
> Is there something wrong with the steps I have taken, appreciate any help.
> 
> Cheers !
> Ayub

Did you consider using XLM-R for your multilingual approach? (Generates language independent embeddings for semantic similarity)<|||||>@timpal0l 
I tested XLM-R for multilingual sentence embeddings.

If used out-of-the-box (without further fine-tuning), the results are really bad, far worse than mBERT (mBERT is also really bad without fine-tuning). 

The vector spaces for XLM-R are not aligned across languages, i.e. the same sentence in two different languages are mapped to completely different points in vector space.

However, when fine-tuned, you can get quite nice results with XLM-R for cross-lingual tasks. Currently I prepare some code + paper + models, which will be release soon in the sentence-transformers repository.

Best
Nils Reimers

<|||||>@nreimers Thanks for you reply!

I see. I have a unlabelled corpus consisting of several languages that I wish to fine tune XLM-R (just update the language models weights to get more domain specific embeddings). Not a down stream task like classification. 

I cant seem to find any example code of doing this, have you managed to do this with XLM-R using HuggingFace? Could you give me any pointers? 

Cheers<|||||>Hi @timpal0l 
I  think this is the file you need
https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py

I haven't tested it by myself.

Best
Nils Reimers<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi @nreimers, I am using sentence transformers for finding stack overflow duplicate questions. I want to train the model from scratch but I am facing some issues. My training set contains  questions and its duplicates only. Is it possible to train the model from this type of training data set.<|||||>Hi @sajit9285 
Just positive examples won't work. You somehow need to teach the network what is similar and what not.

But usually it is not an issue, as getting negative pairs is quite easy. The most simple strategy is just to sample two questions randomly. In 99,9999% of the cases they are non-duplicate and get the negative label.

A better strategy is to use hard negatives, as with the random strategy, your negatives are too easy to spot. One better way would to sample another random question with the same stackoverflow tag and treat it as negative. Or to find a similar question with ElasticSearch BM25 and to assume that it is a negative example<|||||>@nreimers Thanks for your reply. I will try your stated methods. 
I used word2vec averaging method and sentence transformers with pretrained model('bert-base-nli-mean-tokens') for ranking similar questions, and I found word2vec averaging method (for sentence embeddings) performed better. May be the data has lots of tech terms! 
That's why I am thinking of training the model from scratch. <|||||>@sajit9285 Yes, the NLI data sadly does not contain any computer science / programming specific examples, so it does not learn these terms. word2vec is trained on a much wider range of topics, so it has an understanding of programming terms.<|||||>@nreimers So will it work if I trained it from scratch as stated in the that github repo?<|||||>@sajit9285 As always, it depends on the quality of your training data. But I saw quite some good improvements for domain specific terms / sentences if you train it on appropriate training data<|||||>@sajit9285 Is it not better to use the existing weights as a base, rather than train something from scratch? <|||||>@nreimers I will give a try. Thanks  :)<|||||>@timpal0l Yeah ofcourse, anytime they are better than random weights. <|||||>@nreimers You are a beast! A lot of questions I had were addressed on here! <|||||>@nreimers I have tried a lot replacing AllNLI  files with my own dataset files in the same format. I have also changed labels (inside nliReader class' member function named get_labels) from 3 labels(contradiction,neutral,entailment) to two labels (true,false) for my task. But it is still printing those three labels and unable to detect my dataset. I tried a lot but now need your help now. I task that I trying to perform is fine tuning on bert which takes paired para/sentences as input.<|||||>Hello @nreimers. I run all models available in https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/

only using

`model = SentenceTransformer(model_name)`

The model that gave the best results was: distiluse-base-multilingual-cased
The results are very similar than USE (Universal Sentence Enconder). 

My questions are: 

1. How I can improve the results of  distiluse-base-multilingual-cased without clean all weird text cases that I have in dataset?
2. Should I need explore fine-tune parameters? How I can do that?
3. Should I need add more layers after the end layer? What are your suggestions?
4. There are any way to use GloVe pre-trained model with SBERT? If yes, how I can do that?

I want to understand how I can navigate in your beautiful SBERT to do a little/easy modifications that can bring me better results.

Thank you for your help.  

<|||||>> Hi,
> BERT out-of-the-box is not the best option for this task, as the run-time in your setup scales with the number of sentences in your corpus. I.e., if you have 10,000 sentences/articles in your corpus, you need to classify 10k pairs with BERT, which is rather slow.
> 
> A better option is to generate sentence embeddings: Every sentence / article is mapped to a fixed sized vector. You need to map your 3k articles only once to a vector.
> 
> A new query is then also mapped to a vector. In this setup, you only need to run BERT for one sentence (at inference), independent how large your corpus is.
> 
> Then, you can use cosine-similiarity, or manhatten / euclidean distance to find sentence embeddings that are closest = that are the most similar.
> 
> I released today a framework which uses pytorch-transformers for exactly that purpose:
> https://github.com/UKPLab/sentence-transformers
> 
> I also uploaded an example for semantic search, where each sentence in a corpus is mapped to a vector and than cosine-similarity is used to find the most similar sentences / vectors:
> https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py
> 
> Let me know if you have further questions.

Hi there! Phenomenal work! I just had one question, how do transformer encodings (say BERT) compare against encodings from models like Google's Universal Sentence Encoder on a textual semantic similarity task? <|||||>Hi @algoromeo 
Universal Sentence Encoder (USE) spans several different architectures. The USE large is based on transformer networks like  BERT, i.e., the architectures are quite comparable. A big advantage of BERT is the language model pre-training, which induces a lot of information about language in the model. This pre-training is missing in USE.
USE also has CNN networks, which are faster and runtime scales better with the input length. But their performance is usually worse than the transformer based architectures. So you trade speed for lower accurarcy.<|||||>> Hi @algoromeo
> Universal Sentence Encoder (USE) spans several different architectures. The USE large is based on transformer networks like BERT, i.e., the architectures are quite comparable. A big advantage of BERT is the language model pre-training, which induces a lot of information about language in the model. This pre-training is missing in USE.
> USE also has CNN networks, which are faster and runtime scales better with the input length. But their performance is usually worse than the transformer based architectures. So you trade speed for lower accurarcy.

Thank you for your timely and apt reply! Gave me the much needed clarity! Cheers!<|||||>Hi @nreimers , thank you for your detailed explanation on many issues around sentence-bert and semantic textual similarity search. I am currently working on a social science project in which I am trying to measure the "cultural distinctiveness" (basically whether people are different from each other when they comment) of Reddit users based on their comments in certain posts. 

I am thinking of treating all comments of each user as a document. Hopefully, I could obtain document embeddings using sentence transformers. Alternatively, I could use GloVe or Latent Semantic Analysis as embeddings of the document. After that, I am also hoping to compare each individual with the collectives he/she belongs to. So comparing text generated one user against text generated by a group of pre-defined people (and do that iteratively for every user in the dataset). Do you think sentence BERT is a suitable method to embed documents? Could you recommend any work related to the thing I am trying to do, please?  Thank you!<|||||>Hi @SamALIENWARE 
I am afraid that Sentence-BERT is not suitable for that.

BERT (&Co.) have a quadratic runtime and quadratic memory requirement with the text length. I.e., for long documents you would need extremely large memory and have an extremely long runtime. This is why BERT & Co. limit the length for the input document to 512 word pieces, which are about 300 words.

For your purpose I would use avg. GloVe embeddings (which are already implemented in the sentence-transformers project) or LSA/LDA (e.g. from Gensim).

Best
Nils Reimers<|||||>> Hi @SamALIENWARE
> I am afraid that Sentence-BERT is not suitable for that.
> 
> BERT (&Co.) have a quadratic runtime and quadratic memory requirement with the text length. I.e., for long documents you would need extremely large memory and have an extremely long runtime. This is why BERT & Co. limit the length for the input document to 512 word pieces, which are about 300 words.
> 
> For your purpose I would use avg. GloVe embeddings (which are already implemented in the sentence-transformers project) or LSA/LDA (e.g. from Gensim).
> 
> Best
> Nils Reimers

Thanks a million, @nreimers ! I will definitely try your suggestions out. 

I have tried distilled sentence BERT out yesterday. Perhaps its because there aren't that many data (19000+ users in my dataset), the "sentence" embeddings were acquired in a relatively short time period. Then I used k-means clustering on the embeddings and calculated the sum of the distance of each vector to the centroids of the clusters. I am thinking that the larger the sum, the more "distinct" the user's content is since it's semantically far from everyone else's.

So after I got embeddings using GloVe or LSA/LDA, do you think the euclidean distance to k-means centroids is a good representation of semantic textual similarity in a non-pairwise situation (1 vs. many)? Or is it better to stick to cosine similarity (calculate pairwise cosine similarity and then average), as the embedding models are trained using this metric?

Thank you again for your valuable time. I do appreciate it. Have a nice day!<|||||>> @nreimers , I have tried a lot replacing AllNLI files with my own dataset files in the same format. I have also changed labels (inside nliReader class' member function named get_labels) from 3 labels(contradiction,neutral,entailment) to two labels (true,false) for my task. But it is still printing those three labels and unable to detect my dataset. I tried a lot but now need your help now. I task that I trying to perform is fine tuning on bert which takes paired para/sentences as input.

<|||||>@nreimers Brillant Work!!! I just wanted to understand when we are doing evaluation we are using STS Bench Mark but when we have domain-specific data do we still need to STS or we can split our data into test and train and evaluate.  <|||||>Hi @saurabhsaxena86 
No, in that case you don't need STS. If your domain specific data is suitable, you can of course train on that.<|||||>Hey @nreimers, I am bit confused on how to go about training the model from scratch on my dataset. Is there some resource which I can refer to. I am having hard time figuring out how to create the dataloader and train the model on specific data.
<|||||>Hi @saurabhsaxena86 , can you please share the code on how you have trained the model on your domain specific data?
That would be of great help!<|||||>Hi @Shubhamsaboo 
Currently only the these scripts with training examples exists:
https://github.com/UKPLab/sentence-transformers/tree/master/examples/training_transformers

More examples for training will be pushed soon. Further, I currently work on a more extensive documentation.<|||||>HI @nreimers , how can i find average feature vector from the embeddings given by sentence bert for large sentence similarity comparison ?<|||||>Hi @ankitkr3 
Not sure what you mean. But you can use:
https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/util.py#L12

With quite large sets of sentences to compute the cosine sim between all of them. 

This example might also be relevant:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_quora_duplicate_questions/application_duplicate_questions_mining.py<|||||>hi @nreimers I want to calculate the similarity between two large/medium paragraphs. How can i achieve that with the current available models?<|||||>Hi @ankitkr3 
Currently the models are trained and optimized  for sentence length inputs. You can also input longer inputs, up to 510 word pieces (which is the limit for BERT).

One way to compare paragraphs and to get a similarity score would be to use Sentence Mover Distance:
https://www.aclweb.org/anthology/P19-1264v2.pdf

Code available here:
https://github.com/eaclark07/sms

I did not use this code / approach, but I heard that it can produce quite good results when you compare paragraphs with each other.<|||||>@nreimers are you planning to make it available for large sentences or paragraphs ?<|||||>This thread help-me a lot, thanks guys! A question: I have more or less 7000 documents with 300 words each in average and some text from users. My idea is use the text from users as queries to "retrieval" or "ranking" these documents,but I dont know which is the best strategy, STS task or Learn to Rank? Any ideas are welcome. <|||||>Hi @finardi 
Do you have training data in the form (user-query, relevant_doc)? If yes, you can use the MultipleNegativesRankingLoss, which is a learn to rank loss function: https://www.sbert.net/docs/package_reference/losses.html#multiplenegativesrankingloss

<|||||>Hi @nreimers,

For a project I need a good document embeddings for news articles example from cnn.com, what if we use Bert for each paragraph of the article and then average all the embedding vectors, do you think that naive approach could lead to good performance? I am trying to search for similar news covering the same event from different news outlets. Example scenario could be that the company X goes bankrupt, cnn.com will cover the event, CNBC will cover the event, fox.com, and so on, so if the user selects one article the system will show other articles covering the same event, I will pre-filter the news from too far in the future and too far in the past because probably those news are not covering the same event. I have all the news in a database.

Do you think with a simple average word embedding offer good performance in this application? and I forget using Bert? for my application I want the best accuracy I can use GPUs for inference if that is required. 
<|||||>Hi @bushjavier 
I am not sure if BERT / SBERT will work that well for your task.

For documents, the best approach is usually to use TF-IDF / BM25. Often, these documents on the same events have so many word overlaps, that it is quite easy to identify similar documents.

Embedding approaches are more suitable if you want to compare sentences. There, the word overlap can be quite small where TD-IDF / BM25 fails. However, when you average them for larger text, it is quite unclear what the averaged embedding will look like.

Further, you have issues with docs with different lengths.

Assume you have doc A, reporting about event X.
Then you have doc B, reporting about event  X but then also providing background information or reporting about other events. With BM25, this is no issue, it detects that the information of A is include in doc B. With averaged embeddings, the embeddings for doc A and B can be quite different.<|||||>@nreimers thanks for the crystal clear explanation! I will proceed with TF-IDF / BM25.<|||||>HI @nreimers, i've used bert embeddings with success to perform a smart search on a huge software manual. It works pretty well.
I use `'SentenceTransformer('distiluse-base-multilingual-cased')`'. In the corpus i set all the paragraph of the manual and i compare the user query with cosine similarity.
I get great results when the query is a sentence of some words: i.e. "what are supperted servers", "how to upgrade the program", etc. But when the query is a single word, or a single word mispelled or composed by random char (i.e. kjashdjkah) i get false positive often on the first corpus item.
I've made another project in which this issue is more evident: i made a document classificator using an OCR to scan the image and then comparse the results with some sentences. In this case some words read by the OCR are non sense, becouse not all the words are correctly found, and my results a set of words with some strange char. Like i said before in these cases i get false positive, often on the the first corpus item.
Is ther a way to avoid that?

kind regards
Gianluca<|||||>> HI @nreimers, i've used bert embeddings with success to perform a smart search on a huge software manual. It works pretty well.
> I use `'SentenceTransformer('distiluse-base-multilingual-cased')`'. In the corpus i set all the paragraph of the manual and i compare the user query with cosine similarity.
> I get great results when the query is a sentence of some words: i.e. "what are supperted servers", "how to upgrade the program", etc. But when the query is a **single word**, or a single word **mispelled** or composed by random char (i.e. kjashdjkah) i get false positive often on the first corpus item.
> I've made another project in which this issue is more evident: i made a document classificator using an OCR to scan the image and then comparse the results with some sentences. In this case some words read by the OCR are non sense, becouse not all the words are correctly found, and my results a set of words with some strange char. Like i said before in these cases i get false positive, often on the the first corpus item.
> Is ther a way to avoid that?
> 
> kind regards
> Gianluca

Perhaps use word embeddings for single word documents instead of contextualized embeddings such as Bert. And for misspelled words - can't you perform a dictionary checkup to see if the word exists in a vocabulary or not? 
<|||||>
>  can't you perform a dictionary checkup to see if the word exists in a vocabulary or not?

I can't becouse some queries may contains acronyms or other technician abbreviation (i.e. F24) that aren't present in a vocabulary.
Now i'm trying the way suggested from @nreimers some post above: using bm24 and then sentence embeddings to update the score.

<|||||>> > can't you perform a dictionary checkup to see if the word exists in a vocabulary or not?
> 
> I can't becouse some queries may contains acronyms or other technician abbreviation (i.e. F24) that aren't present in a vocabulary.

Cant you build your own word vector model on your domain specific data to learn acronyms and common terms that are not present in a vocabulary. And then allow for the x nearest neighbours? 

<|||||>Man! this Github far more insightful than some of those NLP blogs out there!!! So thanks for the QNA's.
Here articles and sentences are used for semantic search but what about other datasets like well any DB(SQL or JSON)?

Well for an intuitive example suppose I have an object `{Product: Cool Refrigerator, Price: 5000}` and if i type a query like "refrigerators under 5000" the result should be `Result: Cool Refrigerators` there are models out there with similar solutions... but was curios if anyone had good references and solutions. 

THanks<|||||>This Discussion is GOLD!!! 
@nreimers - I salute your patience and knowledge-graphs embedded in your brains :). You answered almost every question with details. I accidentally bumped on this trying to read through issues.

 I usually use Jina, Nboost and Sentence-Transformer depending on the problem statement along with transformers. I learnt a lot from this discussion.  Thanks again to everyone who contributed. This discussion should be part of the Huggingface newsletter.<|||||>Hi, I m trying to merge the output of LDA with BERT. 
@nreimers Could you please route me with right steps on how to merge the output of LDA vectors with BERT for topic modelling task. Which BERT pre-trained model will be best suited for this task. Sentence-transformer or any ?

<|||||>@sarojadevi Never worked with LDA and BERT. Can't help here, sorry.<|||||>@nreimers Hi, I have used sentence-transformer to finetune on my dateset which has Anchor,Positive,Negative.
I use tripletloss as loss function, but after training, the  word embeddings are very similar, so they cosine-similar are all close to 1.
Could tell me how to solve this problem?
By the way, my goal is to find the sentences closest to the input sentence, and all the sentence is about 400 words. Do you have any better suggestions？ <|||||>Hi @zitaozz 
You can try this loss, where you only need anchor and positives (if suitable):
https://www.sbert.net/docs/package_reference/losses.html#multiplenegativesrankingloss

Do you mean word embeddings or sentence embeddings?

<|||||>@nreimers Thanks for you replying! But my dataset seems not suitable for this loss because some positive sentences are similar.
Yes, I mean sentence embeddings.  I don' t know why they are very similar after training.<|||||>Hi @nreimers 
Sorry to barge in on this issue, but i had a query. 
I used the scibert model from the hugging face repository and finetuned it on the SNLI and MNLI datasets using the sentence transformer repository. So I was trying to perform a similarity match between a query sentence and a larger text (title and abstract from arxiv papers). However, I wasn't able to get really good performance (as in the matched samples didn't look too good).

I understand that bert is trained on single sentences and not on multiple. But do you have any recommendations as to how I can improve. Thanks :)<|||||>Hi @MukundVarmaT 
As often, it depends on the training data. SNLI and MNLI are not really good training sets for this.

Have a look at this paper:
https://arxiv.org/abs/2004.07180
<|||||>Hi! I am going through the  exactly same kind of problem as I have around 20M text data.  Please help me with the idea as I have  some code already for it. My idea is to extract the `CLS` token for all the text in the DB and save it in CSV or somewhere else. So when a new text comes in, instead of using the `Cosine Similarity/JAccard/MAnhattan/Euclidean` or other distances, I have to use some approximation like `LSH, ANN (ANNOY, sklearn.neighbor)` or the one given here `faiss` . How can that be done using your library? I have my code as:

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
Using Tensorflow:

import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```

and I can get the `CLS` token as:
```
last_hidden_states = outputs[0]
cls_embedding = last_hidden_states[0][0]
```
Please tell me if it's the right way to use and how can I use any of the `LSH, ANNOT, faiss` or something like that?<|||||>@deshwalmahesh 
As outlined in our paper, produces BERT out of the box rather bad sentence embeddings, especially if you want to use them to find similar news articles:
https://arxiv.org/abs/1908.10084

For finding similar news articles, I can recommend these 3 articles:
https://www.sbert.net/examples/applications/paraphrase-mining/README.html
https://www.sbert.net/examples/applications/parallel-sentence-mining/README.html
https://www.sbert.net/examples/applications/semantic-search/README.html

There, you also find examples how to use approximate nearest neighbor methods, for example, based on faiss and similar<|||||>@nreimers 
> @ggndtes In general BM25 will be really hard to beat on this type of task. See this paper where they compare sentence embeddings with BM25 on an end-to-end retrieval task (given: question, find similar / duplicate questions in a large corpus):
> https://arxiv.org/pdf/1811.08008.pdf

In our experience (building a search engine for a specific domain), sentence-transformers and BM-25 produce different results and answer different use-cases. For short queries, sentence transformers seem to have a hard time formalizing the meaning and resort to finding lexically similar sentences. In this case, BM-25 is probably better, especially if enhanced with synonyms etc.  

However. for longer queries (>=6 words) expressing a compound concept, sentence transformers (with appropriate training data) shine and are able to capture semantically similar yet lexically different sentences. Elasticsearch, on the other hand, failed miserably and managed to capture only part of the meaning. 

Overall, it depends also on the goal. In many cases, the typical information needs are contained within a sentence/short paragraph. There is little meaning to a document besides a collection of such paragraphs (with some dependencies between them). The text is better represented as sentences rather than a long document, and that's where transformer-based semantic encoder can encode meaning well. 

As a side note, there are still substantial challenges with negation and directionality. 

<|||||>@tkorach Thanks for sharing.

I agree, we saw good improvements with dense retriever over the last years. But there are still issues:
- Domain shift: We see good results when there is sufficient training data. However, when you go to a specialized domain (for example publications on COVID 19, as done in TREC COVID), existent dense models perform often rather poorly and are out-performed by BM25. I.e., you need quite some training data to get good results with dense approaches
- In some cases users search for key words, for example, a specific error code  they want further information:  There, dense approach perform often rather poorly. So an interesting area is on how to combine sparse lexical approaches with dense approaches. Especially, when is the user searching for keywords and when not?

<|||||>>  Especially, when is the user searching for keywords and when not?

These could actually represent different information needs. Searching for named entities (individuals, locations, medications etc.) could focus on topical similarity: the entity is atomic, and either exists in the document or not (edge cases like negation can be handled by rule-based approach). For this ask topical occurrence-based methods like BM-25 work well enough and have the advantage of generalizability. 

Compound concepts (e.g. "the patient's nausea improved in response to the treatment") may represent another task. The information need can be described not as finding the most relevant documents, but rather as finding a passage of text that contain the same meaning. In essence, each search query is a binary classification tasks of each and every text in the corpus. In my experience, many customers ask for "search" but are actually looking for information extraction. Semantic similarity serve as a "poor-man's" zero-shot text classifier. 

The use-cases may also differ. In the search projects we did, topical similarity was used to expand their information (e.g. a person travelling to Italy would like to read about Rome), while in semantic search the "search" is used to classify items in a population. This happens frequently when the documents are components of another unit (a medical record, an insurance claim, a legal lawsuit). In professional business settings the second use-case is far more common and the main task revolves, almost always, around classification of cases. This is in contrast to web-search engine, where the document is main unit (we don't search for the web sites themselves but for content within them). 

To make a long story short, we found out that explaining these differences to our users help them understand what can they expect from the algorithm. Experience with Google might push users to downgrade the query to fewer keywords to improve results. With semantic-search (e.g. STS-based encoders), it's the opposite and a more elaborate query will facilitate capturing the meaning and generalizing to similarly-meaning yet lexically-different ones. 
<|||||>@nreimers You had an answer before regarding semantic search, where you suggested doing BM25 to retrieve a first set of documents, and then to re-rank them with sentence embeddings.

1) I cant find that comment anymore.
2) With getting the first subset of the BM25 results, you will only get documents with lexical overlap / direct matches of words. And sure you can probably re-rank them a bit with the semantical information from sentence representations. But this subset throws away alot of documents where there is a high semantic match.. Would you agree?<|||||>@timpal0l There was quite an improvement over the years for semantic search. 

But BM25 is still a really strong system in cases where you have a specialized domain or task and you don't have any training data. Also, BM25 is really strong if you retrieve longer texts (documents). Pre-trained dense retrieval models, for example on MS MARCO or NR, often have issues when you go to specialized domains or when your retrieval task is different than how it is models in MS MARCO / NQ.

Semantic search works well when you have training data or your task&domain is similar to something where you have training data. Also semantic search is quite beneficial when you retrieve short text like sentences.

First retrieving with BM25 and then re-ranking with sentence embeddings is not that sensible. What can make sense is to retrieve with BM25 and to re-rank with a BERT cross-encoder:
https://www.sbert.net/examples/applications/information-retrieval/README.html

Re-ranking with a cross-encoder can substantially boost the performances, but has sadly the downside of being slower and more compute intensive then retrieval with a bi-encoder embedding approach.<|||||>> @timpal0l There was quite an improvement over the years for semantic search.
> 
Agree on that!
>
> But BM25 is still a really strong system in cases where you have a specialized domain or task and you don't have any training data. 
>
Ok, by training data you mean like an annotated corpus in the style of SNLI, STS-B, or similar? But you mention in your sentence-bert paper that you can get around with the self-supervised approach, when you create triplets from wikipedia documents. Doesnt BM25 also require a corpus to calculate the tf-idfs weights from? You think this still is stronger then the self supervised sentence bert with triplet loss (on short sentences)?  
>
>
> Also, BM25 is really strong if you retrieve longer texts (documents). Pre-trained dense retrieval models, for example on MS MARCO or NR, often have issues when you go to specialized domains or when your retrieval task is different than how it is models in MS MARCO / NQ.
>
Agree, I have seen similar results where simpler methods (tf-idf) works better on looong documents, do you think its cause its hard to create a good representation of a long document since the context can change a lot over document? 
> 
> Semantic search works well when you have training data or your task&domain is similar to something where you have training data. Also semantic search is quite beneficial when you retrieve short text like sentences.
> 
> First retrieving with BM25 and then re-ranking with sentence embeddings is not that sensible. What can make sense is to retrieve with BM25 and to re-rank with a BERT cross-encoder:
> https://www.sbert.net/examples/applications/information-retrieval/README.html
> 
Ok, with "sentence embeddings" I meant the BERT bi-encoder (siemiese bert / your sentence bert paper). Right, so you say that by using the cross-encoder, you re-rank all pairs with each other in the form of "sentence_n [SEP] sentence_n+1" style?
> Re-ranking with a cross-encoder can substantially boost the performances, but has sadly the downside of being slower and more compute intensive then retrieval with a bi-encoder embedding approach.

Makes sense its slower than using precalculated embeddings from the bi-encoder.

I might have missed something, but my first questions remains. Dont you possibly throw away matches when getting the first subset based only on the BM25 (lexical / word overlap) match. Sentences with non lexical overlap but with semantic similarity will not be captured to even be re-ranked..?

<|||||>
> > But BM25 is still a really strong system in cases where you have a specialized domain or task and you don't have any training data.
> 
> Ok, by training data you mean like an annotated corpus in the style of SNLI, STS-B, or similar? But you mention in your sentence-bert paper that you can get around with the self-supervised approach, when you create triplets from wikipedia documents. Doesnt BM25 also require a corpus to calculate the tf-idfs weights from? You think this still is stronger then the self supervised sentence bert with triplet loss (on short sentences)?

It depends on your task, so you should have training data similar to your task. BM25 is primarily be used for search (information retrieval), so in the following I will focus on search. If you have a different task, like clustering, paraphrase mining, bitext mining etc., you need a different approach.

For search, you need training data similar to MS MARCO / Natural Questions:  Given a query and a matching (relevant) passage answering the query.

The triplets created from Wikipedia perform extremely badly on search.

We have currently a paper on the way that presents new methods for training embeddings for semantic search if you don't have labeled training data, just a large set of unlabeled text. BM25 is a strong contestant, sometimes much better than dense approach, sometimes worse.
 
BM25 just need unlabeled data to compute idf values. 


> 
> > Also, BM25 is really strong if you retrieve longer texts (documents). Pre-trained dense retrieval models, for example on MS MARCO or NR, often have issues when you go to specialized domains or when your retrieval task is different than how it is models in MS MARCO / NQ.
> 
> Agree, I have seen similar results where simpler methods (tf-idf) works better on looong documents, do you think its cause its hard to create a good representation of a long document since the context can change a lot over document?

I think mapping longer documents to a single, dense vector does not make sense.

If I combine two Wikipedia articles, e.g. Microsoft and Chlorine, then this is not that big of a problem for BM25 because the relevant word overlap between the two articles is rather low. But how should a single dense vector look like? Is it (Vector(Microsoft) + Vector(Chlorine)) / 2
Likely not.


> > Semantic search works well when you have training data or your task&domain is similar to something where you have training data. Also semantic search is quite beneficial when you retrieve short text like sentences.
> > First retrieving with BM25 and then re-ranking with sentence embeddings is not that sensible. What can make sense is to retrieve with BM25 and to re-rank with a BERT cross-encoder:
> > https://www.sbert.net/examples/applications/information-retrieval/README.html
> 
> Ok, with "sentence embeddings" I meant the BERT bi-encoder (siemiese bert / your sentence bert paper). Right, so you say that by using the cross-encoder, you re-rank all pairs with each other in the form of "sentence_n [SEP] sentence_n+1" style?

Training with sentence_n [SEP] sentence_n+1 does not work (neither for bi- nor for cross-encoder).

You train the cross-encoder for:
query [SEP] paragraph

To output a score 0...1 if the paragraph is relevant for query. For this, you need labeled training data which paragraphs are relevant for which queries.

Then you compute the score for all paragraphs: query [SEP] paragraph1, query [SEP] paragraph2, query [SEP] paragraph3...

and re-rank them.



> > Re-ranking with a cross-encoder can substantially boost the performances, but has sadly the downside of being slower and more compute intensive then retrieval with a bi-encoder embedding approach.
> 
> Makes sense its slower than using precalculated embeddings from the bi-encoder.
> 
> I might have missed something, but my first questions remains. Dont you possibly throw away matches when getting the first subset based only on the BM25 (lexical / word overlap) match. Sentences with non lexical overlap but with semantic similarity will not be captured to even be re-ranked..?

Yes, of course you only get matches with word overlap. How bad this is, depends on your task. For many tasks, BM25 can get quite good recall values (e.g. recall@100), but the ranking of BM25 is bad => then re-ranking helps.

If BM25 does not find the relevant hits within the first 100 or 1000 hits, then re-ranking will not bring a benefit. As before and as always: It depends on your specific task


<|||||>> > > But BM25 is still a really strong system in cases where you have a specialized domain or task and you don't have any training data.
> > 
> > 
> > Ok, by training data you mean like an annotated corpus in the style of SNLI, STS-B, or similar? But you mention in your sentence-bert paper that you can get around with the self-supervised approach, when you create triplets from wikipedia documents. Doesnt BM25 also require a corpus to calculate the tf-idfs weights from? You think this still is stronger then the self supervised sentence bert with triplet loss (on short sentences)?
> 
> It depends on your task, so you should have training data similar to your task. BM25 is primarily be used for search (information retrieval), so in the following I will focus on search. If you have a different task, like clustering, paraphrase mining, bitext mining etc., you need a different approach.
> 
> For search, you need training data similar to MS MARCO / Natural Questions: Given a query and a matching (relevant) passage answering the query.
> 
> The triplets created from Wikipedia perform extremely badly on search.
> 
> We have currently a paper on the way that presents new methods for training embeddings for semantic search if you don't have labeled training data, just a large set of unlabeled text. BM25 is a strong contestant, sometimes much better than dense approach, sometimes worse.
> 
> BM25 just need unlabeled data to compute idf values.
> 
> > > Also, BM25 is really strong if you retrieve longer texts (documents). Pre-trained dense retrieval models, for example on MS MARCO or NR, often have issues when you go to specialized domains or when your retrieval task is different than how it is models in MS MARCO / NQ.
> > 
> > 
> > Agree, I have seen similar results where simpler methods (tf-idf) works better on looong documents, do you think its cause its hard to create a good representation of a long document since the context can change a lot over document?
> 
> I think mapping longer documents to a single, dense vector does not make sense.
> 
> If I combine two Wikipedia articles, e.g. Microsoft and Chlorine, then this is not that big of a problem for BM25 because the relevant word overlap between the two articles is rather low. But how should a single dense vector look like? Is it (Vector(Microsoft) + Vector(Chlorine)) / 2
> Likely not.
> 
> > > Semantic search works well when you have training data or your task&domain is similar to something where you have training data. Also semantic search is quite beneficial when you retrieve short text like sentences.
> > > First retrieving with BM25 and then re-ranking with sentence embeddings is not that sensible. What can make sense is to retrieve with BM25 and to re-rank with a BERT cross-encoder:
> > > https://www.sbert.net/examples/applications/information-retrieval/README.html
> > 
> > 
> > Ok, with "sentence embeddings" I meant the BERT bi-encoder (siemiese bert / your sentence bert paper). Right, so you say that by using the cross-encoder, you re-rank all pairs with each other in the form of "sentence_n [SEP] sentence_n+1" style?
> 
> Training with sentence_n [SEP] sentence_n+1 does not work (neither for bi- nor for cross-encoder).
> 
> You train the cross-encoder for:
> query [SEP] paragraph
> 
> To output a score 0...1 if the paragraph is relevant for query. For this, you need labeled training data which paragraphs are relevant for which queries.
> 
> Then you compute the score for all paragraphs: query [SEP] paragraph1, query [SEP] paragraph2, query [SEP] paragraph3...
> 
> and re-rank them.
> 
> > > Re-ranking with a cross-encoder can substantially boost the performances, but has sadly the downside of being slower and more compute intensive then retrieval with a bi-encoder embedding approach.
> > 
> > 
> > Makes sense its slower than using precalculated embeddings from the bi-encoder.
> > I might have missed something, but my first questions remains. Dont you possibly throw away matches when getting the first subset based only on the BM25 (lexical / word overlap) match. Sentences with non lexical overlap but with semantic similarity will not be captured to even be re-ranked..?
> 
> Yes, of course you only get matches with word overlap. How bad this is, depends on your task. For many tasks, BM25 can get quite good recall values (e.g. recall@100), but the ranking of BM25 is bad => then re-ranking helps.
> 
> If BM25 does not find the relevant hits within the first 100 or 1000 hits, then re-ranking will not bring a benefit. As before and as always: It depends on your specific task

Thanks alot! Your answers are as always, very appreciated! Do you have any paper / results, where I can see how much the re-ranking with the cross-encoder increases the search results compared to only using a bi-encoder? :) <|||||>@timpal0l 

I don't have perfect number.

For the TREC 19 Deep Learning dataset, which is based on the MS MARCO dataset, I get the following results (NDCG@10):
BM25 45.5
BM25 retrieve 1k, re-rank with bi-encoder: 68.4  [*]
BM25 retrieve 1k, re-rank with cross-encoder (electra-base): 72


[*] Note, this is re-ranking with a bi-encoder, not retrieval. Retrieval with a bi-encoder yields roughly the same performances for this dataset.

So I would assume that re-ranking with a cross-encoder is about 3 points better than retrieval with a bi-encoder.

For the bi-encoder, I noticed that it can be quite sensitive to noise. For example, for the query:
query: How many people live in London?

It can retrieve a passage like:
passage: It has 2,000 inhabitants.

Here, the cross-encoder easily identifies that the passage is not relevant for the query as there is not information about London in it. The cross-encoder has here a much harder time.
<|||||>What a fantastically informative read this thread has been, thanks to all for contributing to the discussion.

@nreimers  I'm working on a semantic search project using the CORD-19 (covid) corpus of academic papers. Pass in a search phrase to surface a list of most relevant documents, showing an answer excerpt of similar sentence from surfaced doc. 

Ideally I'd like to encode all articles so as not to miss what could be a vital article to the clinical researcher (PubMed prints ~100k articles, preprints ~100K), thats a lot of encoding! Would you advise just to use the abstract, possibly first n and last n sentences from body text, rather than the full article? I amusing the allenai-specter model as sentence transformer for this objective, any comments?

Would you still support BM25 as first pass then re-rank with cross-encoder for this semantic search task?

Regarding fine-tuning BERT-based model (first time fine tuning here) - would I take a split of the cord 19 articles and pass sentence pairs in, as per the [sentence-transformers NLI example](https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/nli/training_nli.py). One thing Im not sure of is that the NLI example uses labeled data I believe, where the cord19 articles are unlabelled , so I just want to fine-tune the model for the domain semantics.

`train_samples.append(InputExample(texts=[row['sentence1'], row['sentence2']], label=label_id))`

Thank-you very much for any guidance.<|||||>Hi @corticalstack 
I think the allenai SPECTER model is not that great for this. It was trained for the task of paper recommendation, i.e. you input a title and an abstract of an paper and the model can return similar papers.

For semantic search, where you input a query or question, I don't expect the model to work that well.

For the encoding: I would split a paper into paragraphs and then encode them individually.

We are currently working on code that will allow to train semantic search models in an unsupervised fashion. Code & documentation will be released this month. It also works quite well for the CORD-19 dataset (as evaluated on the TREC COVID shared task).

But for CORD-19 we saw that BM25 can be hard to beat. The dataset has many specific terminology, like specific names for virus types. Here, lexical search excels as it gives you exactly the papers that talk about this specific type of virus. So I would at least combine semantic search with lexical search. 

The training examples currently available are only when you have labeled data. As mentioned, we will release this month code & docs for training models without labeled data using two different strategies.<|||||>@nreimers Really appreciate the swift feedback and sharing your expertise, thanks! I will take a look at splitting by paragraph, and will most certainly jump on the unsupervised feature as soon as it becomes available. What's the best channel to watch out for it becoming available?

Will focus on BM25, at least in combination with semantic.

Thanks again for all your shared knowledge in this thread, extremely educational. Oh, and the hard work on the repo!<|||||>@corticalstack I will likely post it on twitter once the tutorial for unsupervised semantic search training becomes available:
https://twitter.com/Nils_Reimers

Otherwise, it will appear on www.SBERT.net in a new section on the left side: Unsupervised Training.

Combing semantic search and BM25 results proved quite effective for me. I combined these two via:
final_score = bm25_score_i / max(bm25_score) + lambda*cos_score_i / max(cos_score)

With:
max(bm25_score)  = The BM25 score of your best BM25 hit
max(cos_score) = The cosine similarity score of your best semantic search hit.

Lambda is a factor to balance between BM25 and semantic search, so that you can control which type of search should have the higher impact.

<|||||>Any suggestion on dataset for benchmarking semantic search please - I was thinking MS Marco but [this article says no](https://www.kdnuggets.com/2020/04/ms-marco-evaluate-semantic-search.html)<|||||>The situation is not as bad as described in the blog post. Every dataset for semantic search / information retrieval has a selection bias. Is is impossible to create a dataset without a selection bias, because otherwise you would need to label for every query 8 Million documents (for the MS MARCO case). This is of course not possible.

So as the author writes at the end:
> But despite all those remarks, the most important point here is that if we want to investigate the power and limitations of semantic vectors (pre-trained or not), we should ideally prioritize datasets that are less biased towards term-matching signals. This might be an obvious conclusion, but what is not obvious to us at this moment is where to find those datasets since the bias reported here are likely present in many other datasets due to similar data collection designs.

The described biases are well known and a long recognized problem. But there are no good solutions for it, especially not at scale.

So in conclusion:
- Is the MS MARCO dataset perfect: No
- Is the MS MARCO dataset useful for training and evaluation: Yes, I definitely think so.  It is far better than many other datasets that could be used (STSbenchmark, SQuAD)
- Are there better datasets: The Natural Questions  (NQ) dataset from Google is interesting, but more targeted on answering questions using Wikipedia. So for question-answer retrieval with Wikipedia, I think NQ is better. For broader retrieval, I cannot say yet which is better (MS MARCO / NQ). MS MARCO has also many keyword style queries and non-wikipedia-answerable queries like "weather san diego", which matches broader what people are searching for.
- Will a better score on MS MARCO mean better performance on my task: No. At some point, models will be too specialized on MS MARCO and its selection bias, i.e., they don't get better, they are just better overfitting on MS MARCO selection bias. I sadly don't know at which point (i.e. score range) this will happen. 
- Is a perfect dataset for semantic search possible? Sadly not, creating one even for evaluation is rather expensive. And sadly you always have some selection bias in it<|||||>Grateful for any guidance. Consider the following question asked when performing a semantic search:

_"Which of the current covid19 vaccines in the clinic have reported the highest levels of neutralizing Abs after a single vaccination?_"

The intent behind the question is to get information back about covid19 vaccines generating high levels of antibodies as tested in the clinic (therefore not phase 1/2 trials with mice).  With BERT variants, how may emphasis be placed on certain words. Having tried various search online search tools (e.g. PubMed) and my own draft search tool using off-the-shelf _distilbert-base-nli-stsb-mean-tokens_ and BM25, it seems quite a challenging questions with results not so good.  Thanks for any thoughts.<|||||>@corticalstack 
First note the difference between symmetric and asymmetric semantic search:
https://www.sbert.net/examples/applications/semantic-search/README.html#symmetric-vs-asymmetric-semantic-search

You have an asymmetric use case (answer and query are of different type), but you are using a model that is only suitable for a symmetric case (query and answer have the same amount of content).

Second: Queries like "what is the biggest/largest/highest..." are quite difficult. Sure, if a text mentions it (X is the largest...), then it can be retrieved. But if you only have sentences like (A is 3), (B is 5), (C is 1), then the model has no way to compare these results to find what is the largest. It has no sense for numbers and does not know what is large / the largest or small / the smallest.

<|||||>@nreimers Can you just do a brain dump into a book please? :)

When Googling for "_Symmetric vs asymmetric semantic search_", oddly enough your SBERT site is first hit! With seemingly few very few other relevant hits. Is this distinction so little discussed in the public domain? The "msmarco-distilbert-base-v2" sentence-transformer states training on a **passage retrieval** dataset. Is this the more common phrase under which such an asymetric task is discussed, would really like to read and understand more behind it.

<|||||>@corticalstack 
It is not a common distinction, so no wonder that SBERT is on the first position.

For semantic search you have basically two cases:

**Paraphrase search (or symmetric semantic search)**
E.g., I have a query like "How can I learn Python via the internet" and you want to find a paraphrase like "How to learn Python using the web". Here, query & document are interchangeable, i.e., the relation is symmetric.

**QA Retrieval / Information Retrieval (or asymmetric semantic search)**
You have a short query (like "learning Python") and you want to find a long document providing you with the answer.


If you use a symmetric model for your QA retrieval (asym. search task), you usually end up with bad results.

Passage retrieval mainly describes just that you search for a query and want to retrieve a text passage.
You could also have doc retrieval, you search for a short query and want to retrieve a long document.

I think there is no clear terminological distinction yet between the two outlined cases.  
<|||||>@corticalstack The code for Synthetic Query Generation is available:
https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/query_generation

It can be used when you want to train a semantic search model without having training data. For the given paragraphs in your corpus, it generates queries what people could ask about this paragraph.

It then uses this generated queries and the paragraph to train a model on it.

We saw really good results with this on various datasets:
https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/query_generation

Currently we prepare an extensive publication that explains it in more detail, provides more insights and also shows ways for optimization.<|||||>@nreimers Thanks so much for the update and great work here, shall try it in next week and report back. Look forward to reading more, any approximate expectation for release of publication?<|||||>@corticalstack As we run quite a lot of experiment for various domains and tasks, I expect that it will still take some time until we publish the paper. I think it will be available in about 2 months.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@nreimers finally got around to testing the synthetic code generation, fascinating query generation and working to understand it better. Has the related paper been published?<|||||>Hi @corticalstack 
We did some experiments in: https://arxiv.org/abs/2104.08663

But the longer (and more detailed) paper is sadly not ready yet.<|||||>@nreimers thanks very much for link to paper, will cite in my work<|||||>@nreimers Thanks so much for both the synthetic query generation and BEIR benchmarking utilities, they are fantastic. I've done the following for task of passage (paragraph) retrieval on CORD-19 covid-19 corpus:

- Used synthetic query generation over 10% sample of cord-19 collection, with model/tokenizer _BeIR/query-gen-msmarco-t5-large-v1 for ~600K query/text combinations_ (2 queries generated per paragraph)
- Generated queries to train _distilbert-base-uncased_ by encoding cord-19 article paragraphs, BEIR testing gives **trec_covid ndcg@10 score 0.468**
- Generated queries to train _sentence-transformers/msmarco-distilbert-base-v3_, BEIR testing gives **trec_covid ndcg@10 score 0.466**
- Base _msmarco-distilbert-base-v3_ without training in BEIR testing gives **trec_covid ndcg@10 score 0.477**
- BM25 achieves **trec_covid ndcg@10 score 0.615**

Training done pretty much as per your synthetic generation example:
[query generation train encoder](https://github.com/UKPLab/sentence-transformers/blob/master/examples/unsupervised_learning/query_generation/2_programming_train_bi-encoder.py)

For query generation seq2seq tokenizer Ive set max_length at 300, output query max_length at 64, and epochs for training at 3.

As per your paper [https://arxiv.org/abs/2104.08663](https://arxiv.org/abs/2104.08663), BM25 is hard to beat. But such a gap between BM25 and the neural models. Also identify the trained model performs slightly poorer than trained model.

Would welcome any pointers to try and up trec-covid score for neural models, as this benchmark seems the most comparable to the cord-19 corpus?

**Edit:** mean length of cord-19 article paragraph is 730, and title 100, after running query gen with max_length = 850, **trec_covid ndcg@10 score** up to **0.473** (from 0.466).

**Edit 2** With same cord-19 generated queries training  (input max_length 850, query output max_length 180), and model msmarco-roberta-base-ance-fristp, **trec_covid ndcg@10 score 0.547**. Progress :)

**Edit 3** sampling just 0.5% cord-19 articles (525 articles), now with 4 queries generated per passage (prev=2), with model _castorini/ance-msmarco-passage_, **trec_covid ndcg@10 score 0.553**<|||||>Hi @corticalstack (cc @NThakur20)

We currently also perform some experiments with TREC-COVID-19. Here are some insights:

- When you train with cosine similarity, the model prefers to retrieve publications that have just a title. However, there is a strong annotation bias that such publications were seldom annotated and hence, are automatically marked as not relevant even though they might be relevant. I think a better setup for TREC-COVID-19 is to just focus on publications that have a title and a non-empty abstract
- When you train with dot-product, the model prefers longer documents and hence retrieves more docs with title & abstract, which results in a higher score. However, it is not clear if such a system is really better as annotators did not really annotate publications where the abstract is missing.
- We also observe that dense models retrieve a lot of docs that have not been seen by an annotator and are marked as not relevant. This is problematic, as we don't know if these publications are really irrelevant or not. The fraction of publications that BM25 retrieves and have not been seen by an annotator is a lot smaller, as annotators used BM25 to retrieve the candidates for annotations. So the difference between BM25 and dense models could be primarily due to this difference, that dense model retrieve good hit, but which have not been annotated. We don't know and more extensive annotations would be needed.
<|||||>@nreimers grateful if you can consider the following question: "_Which of the current vaccines in the clinic have reported the highest levels of neutralizing Abs after a single vaccination?_". Appreciate this question could be considered rather ambiguous. However, testing with BM25 and various base/trained dense models (synthetic query generation), top results focus on words like _neutralizing_ , missing the caveat in the question "after a single vaccination" . I'm finding results from models miss opening few words in questions which set the context, such as "Which of the current vaccines".  

Another example is "_Which vaccines have been approved_" with documents scoring highly that contain "_approved_" but not related to vaccines. 

A third example is "_What T-cell epitopes been identified in the Receptor Binding Motif (RBM) region of the S-glycoprotein Receptor Binding Domain (RBD) of the SARS-CoV-2 virus?_" which picks up documents containing _glycoprotein_ but misses the constraint "What t-cell epitopes". Other ranked documents discuss epitopes but not the T-cell types.

In that last example we are only interested in articles that specifically discuss t-cell epitopes and so the question is how we can enforce such constraints with dense models.

It seems that unless there is a Eureka moment hitting a golden paper that directly answers the question, then answers are very much somewhat relevant and can be useful but don't begin to answer the question fully.  Typically , a human researcher, an expert in their field such as biomedical, will search multiple knowledge sources, collating snippets from papers in a spreadsheet, aggregating for a high-quality answer.  I'm interested in how I can refine the dense model to better  capture constraints like "which of the <constraint> have been ......", "what <constraint>.....reports/have been identified ........". Then, aggregating their relevant texts together to more fully answer the question, but that Im sure is another topic and challenge.<|||||>Hi @corticalstack 
In that case I think a cross-encoder is better, that re-ranks your results. See: https://www.sbert.net/examples/applications/retrieve_rerank/README.html

Dense models have limitations that it is really hard to encode multiple constrains from a query into a vector space. Cross-Encoders, on the other hand, are a lot better on these constraint and fine-grade checking.
<|||||>@nreimers @NThakur20

On reading the original [T5](https://arxiv.org/pdf/1910.10683.pdf) paper, as a multi-task model I'm curious to know which task prefix you added to the input sequence and how "interrogatives" were added/coerced to the text output such as "what is", "which of the", "how to"? Is this something you can share? Thanks!

Edit: I assume its the T5 sequence-2-sequence (or text to text) task. Just checking the [docTTTTT](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf) and original [doc2query](https://arxiv.org/pdf/1904.08375.pdf) papers for possible answers, but fascinating that training on collection of query datasets can consistently pose interrogatives.<|||||>@corticalstack 

Hi there, yes, you are right. We use the T5 seq-2-seq model and directly train on the passage-query pairs without adding any prefix. This is also why we don't have any control over what interrogative words (like what, why, etc) the model outputs in the synthetic text. 

The distribution of these words more or less resembles the one present in the training set (therefore, ours resembles the distribution in MSMARCO). One way to control these globally would be to alter the distribution in your train set. I can also think how adding prefixes like, "what: <what query>" and "when: <when query>" in the training set may allow you to do this for individual instances. I haven't experimented with this but similar past experiments suggest this could work.<|||||>Very interesting discussion!

I have a related and hopefully simpler task: 

> finding multiword expressions (MWE) that are most similar to a reference MWE.

MWE can range from bigram short to 5-gram long phrases. Can I please ask if anyone has suggestions on the best approach for this? Would it be simply retraining a word embedding model by adding preprocessed phrases? Or is there a more recent approach that achieves this? 

Much appreciated!  <|||||>You could simply use the library NLTK and their phrases module, then train
your w2v model on your corpus that can contain MWE/phrases ! :)

lör 24 juli 2021 kl. 06:13 skrev Bo Wang ***@***.***>:

> Very interesting discussion!
>
> I have a related and hopefully simpler task: finding multiword expressions
> (MWE) that are most similar to a reference MWE. MWE can range from bigram
> short to 5-gram long phrases. Can I please ask if anyone has suggestions on
> the best approach for this? Would it be simply retraining a word embedding
> model by adding preprocessed phrases? Or is there a more recent approach
> that achieves this?
>
> Much appreciated!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/876#issuecomment-885996543>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSAYJXK33JELKT7XVOVOJ3TZI4UZANCNFSM4IGKGJTQ>
> .
>
<|||||>@timpal0l Thanks! I am actually wondering if there is a more recent approach of embedding MWEs and find the most similar MWEs? I imagine the search space for character-level embeddings is perhaps infinite. Also would it be possible to ask these large LMs such as GPT-3 to complete for example "I feel  ___" providing a reference sentence for example "I feel lack of family support"?<|||||>I am using sentence-bert for creating the embeddings and the faisslibrary for the query search. Does anyone know how I can evaluate the retrieved outputs for a given query on my own dataset?