Update README.md
Browse files
README.md
CHANGED
@@ -32,68 +32,6 @@ NB-SBERT is a [SentenceTransformers](https://www.SBERT.net) model trained on a [
|
|
32 |
|
33 |
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in such a way that similar sentences in different languages should also be close to each other. Ideally, an English-Norwegian sentence pair should have high similarity.
|
34 |
|
35 |
-
## Keyword Extraction
|
36 |
-
The model can be used for extracting keywords from text. The basic technique is to find the words that are most similar to the document. There are various frameworks for doing this. An easy way is to use [KeyBERT](https://github.com/MaartenGr/KeyBERT). This example shows how this can be done.
|
37 |
-
|
38 |
-
```bash
|
39 |
-
pip install keybert
|
40 |
-
```
|
41 |
-
|
42 |
-
```python
|
43 |
-
from keybert import KeyBERT
|
44 |
-
from sentence_transformers import SentenceTransformer
|
45 |
-
sentence_model = SentenceTransformer("NbAiLab/nb-sbert")
|
46 |
-
kw_model = KeyBERT(model=sentence_model)
|
47 |
-
|
48 |
-
doc = """
|
49 |
-
De første nasjonale bibliotek har sin opprinnelse i kongelige samlinger eller en annen framstående myndighet eller statsoverhode.
|
50 |
-
Et av de første planene for et nasjonalbibliotek i England ble fremmet av den walisiske matematikeren og mystikeren John Dee som
|
51 |
-
i 1556 presenterte en visjonær plan om et nasjonalt bibliotek for gamle bøker, manuskripter og opptegnelser for dronning Maria I
|
52 |
-
av England. Hans forslag ble ikke tatt til følge.
|
53 |
-
"""
|
54 |
-
kw_model.extract_keywords(doc, stop_words=None)
|
55 |
-
|
56 |
-
# [('nasjonalbibliotek', 0.5242), ('bibliotek', 0.4342), ('samlinger', 0.3334), ('statsoverhode', 0.33), ('manuskripter', 0.3061)]
|
57 |
-
```
|
58 |
-
|
59 |
-
The [KeyBERT homepage](https://github.com/MaartenGr/KeyBERT) provides other several interesting examples: combining KeyBERT with stop words, extracting longer phrases, or directly producing highlighted text.
|
60 |
-
|
61 |
-
## Topic Modeling
|
62 |
-
To analyse a group of documents and determine the topics, has a lot of use cases. [BERTopic](https://github.com/MaartenGr/BERTopic) combines the power of sentence transformers with c-TF-IDF to create clusters for easily interpretable topics.
|
63 |
-
|
64 |
-
It would take too much time to explain topic modeling here. Instead we recommend that you take a look at the link above, as well as the [documentation](https://maartengr.github.io/BERTopic/index.html). The main adaptation you would need to do to use the Norwegian nb-sbert, is to add the following:
|
65 |
-
|
66 |
-
```python
|
67 |
-
topic_model = BERTopic(embedding_model='NbAiLab/nb-sbert').fit(docs)
|
68 |
-
```
|
69 |
-
|
70 |
-
## Similarity Search
|
71 |
-
Another common use case for a SentenceTransformers model is to find relevant documents or passages of documents given a certain query text. In this scenario, it is pretty common to have a vector database that stores the embedding vectors for all our documents. Then, at runtime, an embedding for the query text is generated and compared efficiently against the vector database.
|
72 |
-
|
73 |
-
While production vector databases exist, a quick way to experiment with them is by using [`autofaiss`](https://github.com/criteo/autofaiss):
|
74 |
-
|
75 |
-
```bash
|
76 |
-
pip install autofaiss sentence-transformers
|
77 |
-
```
|
78 |
-
|
79 |
-
```python
|
80 |
-
from autofaiss import build_index
|
81 |
-
import numpy as np
|
82 |
-
|
83 |
-
from sentence_transformers import SentenceTransformer, util
|
84 |
-
sentences = ["This is a Norwegian boy", "Dette er en norsk gutt", "A red house"]
|
85 |
-
|
86 |
-
model = SentenceTransformer('NbAiLab/nb-sbert')
|
87 |
-
embeddings = model.encode(sentences)
|
88 |
-
index, index_infos = build_index(embeddings, save_on_disk=False)
|
89 |
-
|
90 |
-
# Search for the closest matches
|
91 |
-
query = model.encode(["A young boy"])
|
92 |
-
_, index_matches = index.search(query, 1)
|
93 |
-
print(index_matches)
|
94 |
-
```
|
95 |
-
|
96 |
-
|
97 |
## Embeddings and Sentence Similarity (Sentence-Transformers)
|
98 |
|
99 |
As seen above, using the library [sentence-transformers](https://www.SBERT.net) makes the use of these models quite convenient:
|
@@ -168,6 +106,72 @@ print(scipy_cosine_scores)
|
|
168 |
# This should give 0.8250 in the example above.
|
169 |
|
170 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
171 |
|
172 |
|
173 |
# Evaluation and Parameters
|
|
|
32 |
|
33 |
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in such a way that similar sentences in different languages should also be close to each other. Ideally, an English-Norwegian sentence pair should have high similarity.
|
34 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
## Embeddings and Sentence Similarity (Sentence-Transformers)
|
36 |
|
37 |
As seen above, using the library [sentence-transformers](https://www.SBERT.net) makes the use of these models quite convenient:
|
|
|
106 |
# This should give 0.8250 in the example above.
|
107 |
|
108 |
```
|
109 |
+
## SetFit - Few Shot Classification
|
110 |
+
[SetFit](https://github.com/huggingface/setfit) is a method for using sentence-transformers to solve one of major problem that all NLP researchers are facing: Too few labeled training examples. The 'nb-sbert' can be plugged directly into the SetFit library. Please see [this tutorial](https://huggingface.co/blog/setfit) for how to use this technique.
|
111 |
+
|
112 |
+
|
113 |
+
## Keyword Extraction
|
114 |
+
The model can be used for extracting keywords from text. The basic technique is to find the words that are most similar to the document. There are various frameworks for doing this. An easy way is to use [KeyBERT](https://github.com/MaartenGr/KeyBERT). This example shows how this can be done.
|
115 |
+
|
116 |
+
```bash
|
117 |
+
pip install keybert
|
118 |
+
```
|
119 |
+
|
120 |
+
```python
|
121 |
+
from keybert import KeyBERT
|
122 |
+
from sentence_transformers import SentenceTransformer
|
123 |
+
sentence_model = SentenceTransformer("NbAiLab/nb-sbert")
|
124 |
+
kw_model = KeyBERT(model=sentence_model)
|
125 |
+
|
126 |
+
doc = """
|
127 |
+
De første nasjonale bibliotek har sin opprinnelse i kongelige samlinger eller en annen framstående myndighet eller statsoverhode.
|
128 |
+
Et av de første planene for et nasjonalbibliotek i England ble fremmet av den walisiske matematikeren og mystikeren John Dee som
|
129 |
+
i 1556 presenterte en visjonær plan om et nasjonalt bibliotek for gamle bøker, manuskripter og opptegnelser for dronning Maria I
|
130 |
+
av England. Hans forslag ble ikke tatt til følge.
|
131 |
+
"""
|
132 |
+
kw_model.extract_keywords(doc, stop_words=None)
|
133 |
+
|
134 |
+
# [('nasjonalbibliotek', 0.5242), ('bibliotek', 0.4342), ('samlinger', 0.3334), ('statsoverhode', 0.33), ('manuskripter', 0.3061)]
|
135 |
+
```
|
136 |
+
|
137 |
+
The [KeyBERT homepage](https://github.com/MaartenGr/KeyBERT) provides other several interesting examples: combining KeyBERT with stop words, extracting longer phrases, or directly producing highlighted text.
|
138 |
+
|
139 |
+
## Topic Modeling
|
140 |
+
To analyse a group of documents and determine the topics, has a lot of use cases. [BERTopic](https://github.com/MaartenGr/BERTopic) combines the power of sentence transformers with c-TF-IDF to create clusters for easily interpretable topics.
|
141 |
+
|
142 |
+
It would take too much time to explain topic modeling here. Instead we recommend that you take a look at the link above, as well as the [documentation](https://maartengr.github.io/BERTopic/index.html). The main adaptation you would need to do to use the Norwegian nb-sbert, is to add the following:
|
143 |
+
|
144 |
+
```python
|
145 |
+
topic_model = BERTopic(embedding_model='NbAiLab/nb-sbert').fit(docs)
|
146 |
+
```
|
147 |
+
|
148 |
+
## Similarity Search
|
149 |
+
Another common use case for a SentenceTransformers model is to find relevant documents or passages of documents given a certain query text. In this scenario, it is pretty common to have a vector database that stores the embedding vectors for all our documents. Then, at runtime, an embedding for the query text is generated and compared efficiently against the vector database.
|
150 |
+
|
151 |
+
While production vector databases exist, a quick way to experiment with them is by using [`autofaiss`](https://github.com/criteo/autofaiss):
|
152 |
+
|
153 |
+
```bash
|
154 |
+
pip install autofaiss sentence-transformers
|
155 |
+
```
|
156 |
+
|
157 |
+
```python
|
158 |
+
from autofaiss import build_index
|
159 |
+
import numpy as np
|
160 |
+
|
161 |
+
from sentence_transformers import SentenceTransformer, util
|
162 |
+
sentences = ["This is a Norwegian boy", "Dette er en norsk gutt", "A red house"]
|
163 |
+
|
164 |
+
model = SentenceTransformer('NbAiLab/nb-sbert')
|
165 |
+
embeddings = model.encode(sentences)
|
166 |
+
index, index_infos = build_index(embeddings, save_on_disk=False)
|
167 |
+
|
168 |
+
# Search for the closest matches
|
169 |
+
query = model.encode(["A young boy"])
|
170 |
+
_, index_matches = index.search(query, 1)
|
171 |
+
print(index_matches)
|
172 |
+
```
|
173 |
+
|
174 |
+
|
175 |
|
176 |
|
177 |
# Evaluation and Parameters
|