Update README.md
Browse files
README.md
CHANGED
@@ -27,6 +27,7 @@ in different languages. The finetuning was always performed using English instru
|
|
27 |
- **Language(s) (NLP):** Trained on English, German, French, Spanish.
|
28 |
<!--- **License:** [More Information Needed]-->
|
29 |
<!--- **Finetuned from model [optional]:** [More Information Needed]-->
|
|
|
30 |
|
31 |
|
32 |
### Model Description
|
@@ -270,6 +271,19 @@ from [mteb/scripts/task_selection/europe_tasks.csv at main 路 embeddings-benchma
|
|
270 |
| Pharia-1-Embedding-4608-control | 0.925309 | 0.902113 | 0.937961 | 0.953719 | 0.942352 | 0.945642 | 0.934516 |
|
271 |
| GritLM-7B | 0.934603 | 0.905669 | 0.942364 | 0.962042 | 0.949731 | 0.947428 | 0.940306 |
|
272 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
273 |
## Training Details
|
274 |
|
275 |
### Model architecture
|
@@ -320,27 +334,3 @@ therefore more cost-efficient at inference time.
|
|
320 |
|fr|1.896|2.105|1.836|
|
321 |
|es|1.673|2.030|1.749|
|
322 |
|en|1.633|1.681|1.410|
|
323 |
-
|
324 |
-
|
325 |
-
|
326 |
-
|
327 |
-
|
328 |
-
## Bias, Risks, and Limitations
|
329 |
-
|
330 |
-
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
331 |
-
|
332 |
-
[More Information Needed]
|
333 |
-
|
334 |
-
### Recommendations
|
335 |
-
|
336 |
-
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
|
337 |
-
|
338 |
-
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
|
339 |
-
|
340 |
-
|
341 |
-
**BibTeX:**
|
342 |
-
|
343 |
-
[More Information Needed]
|
344 |
-
|
345 |
-
**APA:**
|
346 |
-
|
|
|
27 |
- **Language(s) (NLP):** Trained on English, German, French, Spanish.
|
28 |
<!--- **License:** [More Information Needed]-->
|
29 |
<!--- **Finetuned from model [optional]:** [More Information Needed]-->
|
30 |
+
- **USP:** Model exhibits superior quality in pure cross-lingual tasks for (German, English, French & Spanish pairings, see evaluation below)
|
31 |
|
32 |
|
33 |
### Model Description
|
|
|
271 |
| Pharia-1-Embedding-4608-control | 0.925309 | 0.902113 | 0.937961 | 0.953719 | 0.942352 | 0.945642 | 0.934516 |
|
272 |
| GritLM-7B | 0.934603 | 0.905669 | 0.942364 | 0.962042 | 0.949731 | 0.947428 | 0.940306 |
|
273 |
|
274 |
+
|
275 |
+
#### Evaluations on cross-lingual capabilities
|
276 |
+
There are important use cases where one wants to retrieve multiple documents on a topic or answering questions that are formulated in a
|
277 |
+
different language than the query. This increases recall and information retrieval coverage. For testing on cross-lingual capabilities
|
278 |
+
evaluated Pharia-1-Embedding-4608-control and GritLM on the MLQA-V1 datasets (Facebook) for German/English and English/Spanish language pairings.
|
279 |
+
For German/French we used the CLSD-WMT19 dataset providing correct and adversarial translations of a sentence in the corresponding pair language.
|
280 |
+
|
281 |
+
|Model Name |MLQA-V1 Ger/Eng (2000 samples)| MLQA-V1 Eng/Esp (2000 samples)| CLSD-WMT19 (2900 samples)|
|
282 |
+
|:-----------------------------:|:----------------------------:|:-----------------------------:|:------------------------:|
|
283 |
+
|Pharia-1-Embedding-4608-control|79.5% |78.5% |95.1% |
|
284 |
+
|GritLM-7B |73.4% |73.9% |94.2% |
|
285 |
+
|
286 |
+
|
287 |
## Training Details
|
288 |
|
289 |
### Model architecture
|
|
|
334 |
|fr|1.896|2.105|1.836|
|
335 |
|es|1.673|2.030|1.749|
|
336 |
|en|1.633|1.681|1.410|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|