peralp24 commited on
Commit
b38a29b
verified
1 Parent(s): d5e5c92

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -24
README.md CHANGED
@@ -27,6 +27,7 @@ in different languages. The finetuning was always performed using English instru
27
  - **Language(s) (NLP):** Trained on English, German, French, Spanish.
28
  <!--- **License:** [More Information Needed]-->
29
  <!--- **Finetuned from model [optional]:** [More Information Needed]-->
 
30
 
31
 
32
  ### Model Description
@@ -270,6 +271,19 @@ from [mteb/scripts/task_selection/europe_tasks.csv at main 路 embeddings-benchma
270
  | Pharia-1-Embedding-4608-control | 0.925309 | 0.902113 | 0.937961 | 0.953719 | 0.942352 | 0.945642 | 0.934516 |
271
  | GritLM-7B | 0.934603 | 0.905669 | 0.942364 | 0.962042 | 0.949731 | 0.947428 | 0.940306 |
272
 
 
 
 
 
 
 
 
 
 
 
 
 
 
273
  ## Training Details
274
 
275
  ### Model architecture
@@ -320,27 +334,3 @@ therefore more cost-efficient at inference time.
320
  |fr|1.896|2.105|1.836|
321
  |es|1.673|2.030|1.749|
322
  |en|1.633|1.681|1.410|
323
-
324
-
325
-
326
-
327
-
328
- ## Bias, Risks, and Limitations
329
-
330
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
331
-
332
- [More Information Needed]
333
-
334
- ### Recommendations
335
-
336
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
337
-
338
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
339
-
340
-
341
- **BibTeX:**
342
-
343
- [More Information Needed]
344
-
345
- **APA:**
346
-
 
27
  - **Language(s) (NLP):** Trained on English, German, French, Spanish.
28
  <!--- **License:** [More Information Needed]-->
29
  <!--- **Finetuned from model [optional]:** [More Information Needed]-->
30
+ - **USP:** Model exhibits superior quality in pure cross-lingual tasks for (German, English, French & Spanish pairings, see evaluation below)
31
 
32
 
33
  ### Model Description
 
271
  | Pharia-1-Embedding-4608-control | 0.925309 | 0.902113 | 0.937961 | 0.953719 | 0.942352 | 0.945642 | 0.934516 |
272
  | GritLM-7B | 0.934603 | 0.905669 | 0.942364 | 0.962042 | 0.949731 | 0.947428 | 0.940306 |
273
 
274
+
275
+ #### Evaluations on cross-lingual capabilities
276
+ There are important use cases where one wants to retrieve multiple documents on a topic or answering questions that are formulated in a
277
+ different language than the query. This increases recall and information retrieval coverage. For testing on cross-lingual capabilities
278
+ evaluated Pharia-1-Embedding-4608-control and GritLM on the MLQA-V1 datasets (Facebook) for German/English and English/Spanish language pairings.
279
+ For German/French we used the CLSD-WMT19 dataset providing correct and adversarial translations of a sentence in the corresponding pair language.
280
+
281
+ |Model Name |MLQA-V1 Ger/Eng (2000 samples)| MLQA-V1 Eng/Esp (2000 samples)| CLSD-WMT19 (2900 samples)|
282
+ |:-----------------------------:|:----------------------------:|:-----------------------------:|:------------------------:|
283
+ |Pharia-1-Embedding-4608-control|79.5% |78.5% |95.1% |
284
+ |GritLM-7B |73.4% |73.9% |94.2% |
285
+
286
+
287
  ## Training Details
288
 
289
  ### Model architecture
 
334
  |fr|1.896|2.105|1.836|
335
  |es|1.673|2.030|1.749|
336
  |en|1.633|1.681|1.410|