NLP Bert model to to calculate text similarity, same sentence but not close similarity
Dear expert here:
I have a simple program to calculate text similarity. The program is copied from internet. Initially, I have a list of sentences or stored in db and fetched from db, then
1) I make the list vectorizered and convered into bert model's vector, then using cosine to calculate the similarity. 373
2) 2nd time, I instantiate a new vectorizer and vectorizer only one sentence and then compare against the first time bert model's vector, the result is different from 1st one. 3) 3rd time, I using original vectorizer and vectorizered same sentence again, I notice the result is still different.
Conclusion，one has to vectorizer sentence one by one, not a list. Why???
``` from scipy import spatial from sent2vec.vectorizer import Vectorizer
sentences = [ "This is an awesome book to learn NLP.", "This is an awesome book to learn NLP.", "This is a great book to learn NLP", "This is a great book to learn NLP bert model", "DistilBERT is an amazing NLP model.", "We can interchangeably use embedding, encoding, or vectorizing.", "How about a chinese dinner? this shall be close to 1 or close to 0, I suppose it is close to 1, i.e. the distance is very big" ]
vectorizer = Vectorizer() vectorizer.bert(sentences) vectors_bert = vectorizer.vectors
dist_1 = spatial.distance.cosine(vectors_bert[0], vectors_bert[1]) dist_2 = spatial.distance.cosine(vectors_bert[0], vectors_bert[2]) dist_3 = spatial.distance.cosine(vectors_bert[0], vectors_bert[3]) dist_4 = spatial.distance.cosine(vectors_bert[0], vectors_bert[4]) dist_5 = spatial.distance.cosine(vectors_bert[0], vectors_bert[5]) dist_6 = spatial.distance.cosine(vectors_bert[0], vectors_bert[6]) print('dist_1: {0}, dist_2: {1}, dist_3: {2}, dist_4: {3}, dist_5: {4}, dist_6: {5}'.format(dist_1, dist_2, dist_3, dist_4, dist_5, dist_6))
test2 = [] test2.append(sentences[0]) vectorizer2 = Vectorizer() vectorizer2.bert(test2) vectors_bert2 = vectorizer2.vectors
dist_same_string_different_vectorizer = spatial.distance.cosine(vectors_bert[0], vectors_bert2[0]) print("dist same string different vectorizer is {0}".format(dist_same_string_different_vectorizer))
test3 = [] test3.append(sentences[0]) vectorizer.bert(test3) vectors_bert3 = vectorizer.vectors print("length of vector_bert3 is {0}".format(len(vectors_bert3))) dist_same_string_same_vectorizer_different_time = spatial.distance.cosine(vectors_bert[0], vectors_bert3[0]) print("dist same string same vectorizer different time is {0}".format(dist_same_string_same_vectorizer_different_time)) ```
/2/result
dist_1: 0.0, dist_2: 0.039356231689453125, dist_3: 0.06737476587295532, dist_4: 0.046457767486572266, dist_5: 0.12459474802017212, dist_6: 0.1990041732788086
dist same string different vectorizer is 0.21568220853805542
length of vector_bert3 is 1
dist same string same vectorizer different time is 0.21568220853805542
Process finished with exit code 0 392