readme: ---
  language: fr
  datasets:
  - FrenchMedMCQA
  license: apache-2.0
  model-index:
  - name: qanastek/FrenchMedMCQA-BART-base-Wikipedia-BM25
    results:
    - task:
        type: question-answering
        name: Question Answering
      dataset:
        name: FrenchMedMCQA
        type: FrenchMedMCQA
        config: FrenchMedMCQA
        split: validation
      metrics:
      - name: Exact Match
        type: exact_match
        value: 18.64
        verified: true
      - name: Hamming Score
        type: hamming score
        value: 38.72
        verified: true
  widget:
  - text: "Parmi les bactéries suivantes, laquelle est un agent habituel de méningite néonatale? \n (A) Clostridium tetani (B) Salmonella sérovar Typhimurium (C) Streptococcus agalactiae . (D) Haemophilus influenzae (E) Vibrio cholerae\nLes premiers cas d’infection néonatale à streptocoques du groupe B ont été décrits par Eickhoff en 1964.Cette bactérie est aussi responsable d'infection chez les personnes âgées."
  ---
  
  # FrenchMedMCQA : Multiple-choice question answering on pharmacology exams using BART-base, Wikipedia external knowledge and BM25 retriever
  
  - Corpora: [FrenchMedMCQA](https://github.com/qanastek/FrenchMedMCQA)
  - Model: [BART Base](https://huggingface.co/facebook/bart-base)
  - Number of Epochs: 30
  
  **People Involved**
  
  * [Yanis LABRAK](https://www.linkedin.com/in/yanis-labrak-8a7412145/) (1)
  * [Adrien BAZOGE](https://fr.linkedin.com/in/adrien-bazoge-6b511b145) (2)
  * [Richard DUFOUR](https://cv.archives-ouvertes.fr/richard-dufour) (2)
  * [Béatrice DAILLE](https://scholar.google.com/citations?user=-damXYEAAAAJ&hl=fr) (2)
  * [Pierre-Antoine GOURRAUD](https://fr.linkedin.com/in/pierre-antoine-gourraud-35779b6) (3)
  * [Emmanuel MORIN](https://scholar.google.fr/citations?user=tvTEtM0AAAAJ&hl=fr) (2)
  * [Mickael ROUVIER](https://scholar.google.fr/citations?user=0fmu-VsAAAAJ&hl=fr) (1)
  
  **Affiliations**
  
  1. [LIA, NLP team](https://lia.univ-avignon.fr/), Avignon University, Avignon, France.
  2. [LS2N, TALN team](https://www.ls2n.fr/equipe/taln/), Nantes University, Nantes, France.
  3. [CHU Nantes](https://www.chu-nantes.fr/), Nantes University, Nantes, France.
  
  ## Demo: How to use in HuggingFace Transformers
  
  Requires [Transformers](https://pypi.org/project/transformers/): ```pip install transformers```
  
  ```python
  from datasets import load_dataset
  from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
  
  path_model = "qanastek/FrenchMedMCQA-BART-base-Wikipedia-BM25"
  
  tokenizer = AutoTokenizer.from_pretrained(path_model)
  model = AutoModelForSequenceClassification.from_pretrained(path_model)
  
  pipeline = pipeline(task="summarization", model=model, tokenizer=tokenizer) # CPU
  
  dataset  = load_dataset("qanastek/FrenchMedMCQA")["test"]
  
  for e in dataset:
      prediction = pipeline(source, truncation=True, max_length=900)[0]["summary_text"]
  ```
  
  Output:
  
  ![Preview Output](preview.PNG)
  
  ## Training data
  
  The questions and their associated candidate answer(s) were collected from real French pharmacy exams on the remede website. Questions and answers were manually created by medical experts and used during  examinations. The dataset is composed of 2,025 questions with multiple answers and 1,080 with a single one, for a total of 3,105 questions. Each instance of the dataset contains an identifier, a question, five options (labeled from A to E) and correct answer(s). The average question length is 14.17 tokens and the average answer length is 6.44 tokens. The vocabulary size is of 13k words, of which 3.8k are estimated medical domain-specific words (i.e. a word related to the medical field). We find an average of 2.49 medical domain-specific words in each question (17 % of the words) and 2 in each answer (36 % of the words). On average, a medical domain-specific word is present in 2 questions and in 8 answers.
  
  | # Answers | Training | Validation | Test | Total |
  |:---------:|:--------:|:----------:|:----:|:-----:|
  |     1     |    595   |     164    |  321 | 1,080 |
  |     2     |    528   |     45     |  97  |  670  |
  |     3     |    718   |     71     |  141 |  930  |
  |     4     |    296   |     30     |  56  |  382  |
  |     5     |    34    |      2     |   7  |   43  |
  |   Total   |   2171   |     312    |  622 | 3,105 |
  
  ## Evaluation results
  
  The test corpora used for this evaluation is available on [Github](https://github.com/qanastek/FrenchMedMCQA).
  
  |   Architecture   | Hamming |  EMR  | Hamming |  EMR  | Hamming |  EMR  | Hamming |  EMR  | Hamming |  EMR  |
  |:----------------:|:-------:|:-----:|:-------:|:-----:|:-------:|:-----:|:-------:|:-----:|:-------:|:-----:|
  |   BioBERT V1.1   |  36.19  | 15.43 |  **38.72**  | 16.72 |  33.33  | 14.14 |  35.13  | 16.23 |  34.27  | 13.98 |
  |    PubMedBERT    |  33.98  | 14.14 |  34.00  | 13.98 |  35.66  | 15.59 |  33.87  | 14.79 |  35.44  | 14.79 |
  |  CamemBERT-base  |  36.24  | 16.55 |  34.19  | 14.46 |  34.78  | 15.43 |  34.66  | 14.79 |  34.61  | 14.95 |
  | XLM-RoBERTa-base |  37.92  | 17.20 |  31.26  | 11.89 |  35.84  | 