Update README.md
Browse files
README.md
CHANGED
@@ -30,13 +30,35 @@ widget:
|
|
30 |
# NB-SBERT
|
31 |
[Sentence-transformers](https://www.SBERT.net) model trained on the [machine translated mnli-dataset](https://huggingface.co/datasets/NbAiLab/mnli-norwegian) starting from [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base).
|
32 |
|
33 |
-
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model in different framework. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in a way where we try to keep this distance also between languages. Ideally
|
34 |
|
35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
36 |
|
37 |
-
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
38 |
|
39 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
40 |
pip install -U sentence-transformers
|
41 |
```
|
42 |
|
@@ -65,7 +87,7 @@ print(scipy_cosine_scores)
|
|
65 |
|
66 |
|
67 |
|
68 |
-
|
69 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
70 |
|
71 |
```python
|
|
|
30 |
# NB-SBERT
|
31 |
[Sentence-transformers](https://www.SBERT.net) model trained on the [machine translated mnli-dataset](https://huggingface.co/datasets/NbAiLab/mnli-norwegian) starting from [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base).
|
32 |
|
33 |
+
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model in different framework. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in a way where we try to keep this distance also between languages. Ideally an English-Norwegian sentence pair should have high similarity.
|
34 |
|
35 |
+
## Keyword Extraction
|
36 |
+
The model can be used for extracting keywords from the text. The basic technique is to find the words that are most similar to the document. There are various frameworks for doing this. An easy way is to use [keyBERT](https://github.com/MaartenGr/KeyBERT). This example shows how this can be used.
|
37 |
+
|
38 |
+
```bash
|
39 |
+
pip install keybert
|
40 |
+
```
|
41 |
+
|
42 |
+
```bash
|
43 |
+
from keybert import KeyBERT
|
44 |
+
from sentence_transformers import SentenceTransformer
|
45 |
+
sentence_model = SentenceTransformer("NbAiLab/nb-sbert")
|
46 |
+
kw_model = KeyBERT(model=sentence_model)
|
47 |
+
|
48 |
+
doc = """
|
49 |
+
Nasjonalbiblioteket er et norsk bibliotek som «bevarer fortiden for fremtiden» gjennom en nasjonal kultur- og kunnskapsbase. Rollen som nasjonalbibliotek innebærer blant annet administrering av pliktavleveringsloven som sier at alle dokument som produseres i Norge skal pliktavleveres til Nasjonalbiblioteket. Dette innebærer bøker, aviser, tidsskrifter, digitale dokumenter, film, video, fotografi, kart, kringkasting, lydbøker, musikk, notetrykk, postkort, plakater, småtrykk og teatermateriale. Alt stilles til rådighet for publikum i Nasjonalbibliotekets publikumslokaler i Oslo.
|
50 |
+
"""
|
51 |
+
kw_model.extract_keywords(doc, stop_words=None)
|
52 |
|
|
|
53 |
|
54 |
```
|
55 |
+
|
56 |
+
|
57 |
+
## Embeddings and Sentence Similarity (Sentence-Transformers)
|
58 |
+
|
59 |
+
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
60 |
+
|
61 |
+
```bash
|
62 |
pip install -U sentence-transformers
|
63 |
```
|
64 |
|
|
|
87 |
|
88 |
|
89 |
|
90 |
+
## Embeddings and Sentence Similarity (HuggingFace Transformers)
|
91 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
92 |
|
93 |
```python
|