readme: ---
  language: de
  datasets:
  - germandpr-beir
  pipeline_tag: sentence-similarity
  tags:
  - information retrieval
  - ir
  - documents retrieval
  - passage retrieval
  - beir
  - benchmark
  - qrel
  - sts
  - semantic search
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - transformers
  task_categories:
  - sentence-similarity
  - feature-extraction
  - text-retrieval
  - other
  task_ids:
  - document-retrieval
  ---
  
  # Model card for PM-AI/bi-encoder_msmarco_bert-base_german
  
  ## Model summary
  This model can be used for **semantic search** and **documents retrieval** to find relevant passages based on a query.
  It was trained on a machine translated **MSMARCO dataset** for _german_ with **hard negatives** and **Margin MSE loss**.
  Combining these elements results in a SOTA transformer for asymmetric search.
  Details are presented below.
  
  The model can be easily used with [Sentence Transformer](https://github.com/UKPLab/sentence-transformers) library.
  
  ## Training Data
  The model is based on training with samples from **[MSMARCO Passage Ranking](https://microsoft.github.io/msmarco/#ranking)** dataset.
  It contains about 500.000 questions and 8.8 million passages.
  The training objective is to identify the relevant passages or answers for an input question.
  In terms of content, the texts deal with diverse domains.
  Questions are available as sentences but also keyword-based variants can be found.
  Consequently, models trained on MSMARCO can be used in a variety of domains.
  
  The dataset was originally published in English, but has been translated into other languages by researchers with the help of machine translation.
  To be more specific, **"[mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://arxiv.org/abs/2108.13897)"** is used, which contains 13 Google based translations, German is one of them.
  
  An existing script from the [BEIR framework](https://openreview.net/forum?id=wCu6T5xFjeJ) was used for the training - more details will follow later.
  This script requires a certain structure for parsing the training data, which is not fulfilled by [unicamp-dl/mmarco](https://huggingface.co/datasets/unicamp-dl/mmarco).
  UKP Lab (TU Darmstadt) created an appropriately [processed mmarco](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/) version, that cannot be used, because it contains outdated texts from an older version of unicamp-dl/mmarco (it us using a MarianNMT-based translation instead of Google)
  Since the textual quality of the older version is poorer, a workaround is necessary in order to be able to use the training data translated by Google.
  
  BEIR requires the following structure for the training data when using the `GenericDataLoader`:
  - `corpus.jsonl`: contains one JSON string per line with `_id`, `title` and `text`. 
    - Example: `{"_id": "1234", "title": "", "text": "some text"}`
  - `queries.jsonl` an `_id` and a `text` is required per JSON string per line.
    - Example: `{"_id": "5678", "text": "a question?"}`
  - `qrels/dev.tsv`: represents the relation between question (`query-id`) and correct answer (`corpus-id`). The `score` column is mandatory, but always 1
    - Example:  `1234	5678	1`
  - `qrels/train.tsv`: Structure is identical to `dev.tsv`
  
  Note: Instead of using `GenericDataLoader`, it is also possible to use `HFDataLoader`.
  In this case, a [Huggingface dataset](https://huggingface.co/docs/datasets/index) is loaded directly, i.e. no individual files have to be created manually.
  Nevertheless, this approach also requires a specific structure.
  Two dataset repositories are needed: one for `queries` and `corpus` and another for `qrels`. 
  In addition, specific subset names must be defined.
  Overall, the effort is more extensive, because new datasets have to be created (and uploaded to Huggingface Datasets).
  The variant presented here uses existing datasets that are only minimally adapted and thus offer maximum compatibility. 
  
  The custom-made script [mmarco_beir.py](https://huggingface.co/PM-AI/bi-encoder_msmarco_bert-base_german/blob/main/mmarco_beir.py) contains all necessary adaptations for BEIR compatibility.
  It can be applied to all 14 languages of the mmarco dataset so that corresponding models can be trained comfortably.
  
  ```python
  # mmarco_beir.py
  
  import json
  import os
  import urllib.request
  
  import datasets
  
  # see https://huggingface.co/datasets/unicamp-dl/mmarco for supported languages
  LANGUAGE = "german"
  # target directory containin BEIR (https://github.com/beir-cellar/beir) compatible files
  OUT_DIR = f"mmarco-google/{LANGUAGE}/"
  
  os.makedirs(OUT_DIR, exist_ok=True)
  
  # download google based collection/corpus translation of msmarco and write corpus.jsonl for BEIR compatibility
  mmarco_ds = datasets.load_dataset("unicamp-dl/mmarco", f"collection-{LANGUAGE}")
  with open(os.path.join(OUT_DIR, "corpus.jsonl"), "w", encoding="utf-8") as out_file:
      for entry in mmarco_ds["collection"]:
          entry = {"_id": str(entry["id"]), "title": "", "text": entry["text"]}
          out_file