transformers pytorch camembert token-classification medical fr license:mit autotrain_compatible endpoints_compatible region:us
---
language:
- fr
library_name: transformers
license: mit
metrics:
- precision
- recall
- f1
pipeline_tag: token-classification
tags:
- medical
widget:
- text: absence de signe d'anomalie des petites voies aériennes, notamment pas de
    signe de piégeage.
---

# Neg-CamemBERT-bio: Language model for negation detection in radiological and other clinical texts for the French language.

Neg-CamemBERT-bio is a refined version of the transformer-based [**CamemBERT-bio-base model**](https://huggingface.co/almanach/camembert-bio-base), fine-tuned for the recognition of negations in clinical texts.
Neg-CamemBERT-bio automatically detects both the negation cues and their scope. 
A public model is available for negation recognition in biomedical texts, along with two additional models that are kept private since trained with potentially sensitive data from Lyon University Hospital (Hospices Civils de Lyoon, HCL). 
— one dedicated to radiology reports and the other designed more broadly for various medical texts.

## Model Details



1- Neg-CamemBERT-bio: Fine-tuning of the CamemBERT-bio-base model for negation recognition in biomedical texts in French.

2- Neg-Radio-CamemBERT-bio:  Fine-tuning of the CamemBERT-bio-base model for negation recognition in anonymized texts extracted from French-written thoracic CT scans
provided by the Radiology Department of the Hospices Civils of Lyon.

3- Neg-Medical-CamemBERT-bio:  Fine-tuning of the CamemBERT-bio-base model for negation recognition in clinical texts across various medical domains 
(Radiology, Biomedical, Medical literature,...) in French.



| Model name | Type | Corpus train | Number of sentences  | Negative sentences |
| :----------- | :---- | :---- | :---------------: | :-------------------: |
| `Neg-CamemBERT-bio` | public| ESSAI + CAS | 11 037  | 1 812  |2 009 |
| `Neg-Radio-CamemBERT-bio`| privite| RADIO  | 10 798  | 2 321   | 2 762|
| `Neg-Medical-CamemBERT-bio` | privite | RADIO + ESSAI + CAS + QUAERO| 21 956  | 4 244|


### Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

| **Corpus** | **Details**                                                        |**Licence**|
|------------|--------------------------------------------------------------------|--------------|
| RADIO    | Corpus extracted from thoracic scan reports provided by  the Radiology Department of Lyon University Hospital (Hospices Civils de Lyon, HCL).| Private corpus|
| ESSAI      | ESSAI Clinical is a freely-available corpus that contains clinical trial protocols in French language collected from the registry of the National Cancer Institute (data available [here](https://clementdalloux.fr/?page_id=28)). | CC BY-NC-SA 4.0 DEED |
|  CAS | CAS is a freely-available corpus in French containing clinical cases as published in scientific, legal, or educational literature (data available [here](https://clementdalloux.fr/?page_id=28)).| CC BY-NC-SA 4.0 DEED |
| QUAERO|   QUAERO is freely-available corpus that contains a vast amount of information in the biomedical field and is available in the form of free-text in natural language (data available [here](https://quaerofrenchmed.limsi.fr/)). |  GNU Free Documentation License  |                                                            | 413 M      |

The training dataset distinguishes between the beginning, inside, and end of each negation entity using a BIO annotation scheme.

Abbreviation|Description
-|-
O     |Outside of a named entity, represents the affirmative part of the sentence.
B-cue |Beginning of the negation cue.
I-cue | Inside of the negation cue.
B-scope |Beginning of the negation scope.
I-scope |Inside of the negation scope.

### Fine-tuning

The CamemBERT-bio-base model, along with its tokenizer, was fine-tuned for the token classification task in order to identify negation cues and their scope.
We used the Adam optimizer with a learning rate of 5e−5, and a batch size of 16. 
The maximum length limit was set to 512 tokens. 
The model was trained using cross-validation with 10 folds (90% training / 10% validation). For each fold, the model was trained for 25 epochs.
We eventually selected the best model out of the 10 folds.

### Evaluation

To evaluate the performance of the model and quantify the results, we used the following metrics: 
precision (P), recall (R), and the F1 score (F1). The scores were measured using the seqeval tool.

### Results 

Validation was performed on a 10% sample of sentences from the training set for each model.

| Model's | Validation Dataset | Metrcis Score | Entity 1: cue  | Entity 2: scope  |
| :----------- | :------:| :---------------:|:---------------: | :---------------: |
| `Neg-CamemBERT-bio`| 10% (ESSAI+CAS)  | P    | 95.70 ± 0.94| 86.43 ± 1.12|
|              |         | R | 97.70 ± 0.56| 85.55 ± 1.33|
|              |         | F1 | 96.68 ± 0.46 |87.46 ± 0.93 |
| `Neg-Radio-CamemBERT-bio`     | 10% (RADIO)  | P    | 99.35 ± 0.24 | 94.19 ± 0.94|
|              |        | R    |99.37 ± 0.31  |94.80 ± 0.84  |
|              |        | F1  |99.36 ± 0.25   |94.49 ± 0.80 |
| `Neg-Medical-CamemBERT-bio`| 10%(NegRADIO + ESSAI + CAS + QUAERO)| P    | 97.75 ± 0.45 |90.48 ± 0.74 |
|              |         | R    |98.67 ± 0.20  |91.34 ± 0.60  |
|              |         | F1    |98.20 ± 0.21   | 90.90 ± 0.585|

### Model Description


- **Developed by:** Salim SADOUNE, Antoine Richard, François Talbot,Thomas Guyet, Loic Boussel and Hugues Berry
- **Model type:** NER 
- **Language(s) (NLP):** French
- **License:**  MIT
- **Finetuned from model:** See the [**CamemBERT-bio-base model**](https://huggingface.co/almanach/camembert-bio-base) for more information on this model.


### Direct Use

You can use the public model with Transformers *pipeline* for NER. 

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("aistrosight/Neg-CamemBERT-bio")
model = AutoModelForTokenClassification.from_pretrained("aistrosight/Neg-CamemBERT-bio")

NegNamedEntityRecogniser = pipeline(task="token-classification", model=model , tokenizer = tokenizer,   aggregation_strategy="simple")

text = ["absence de signe d'anomalie des petites voies aériennes, notamment pas de signe de piégeage."]

sample =NegNamedEntityRecogniser(text)
print(sample)

```

You can visualize the cue and the scope of the negation with the library spacy in a Jupyter notebook.

```python
import spacy
def visualize(sample, text):

    colors = {'scope_neg': "#61ffab", "cue_neg": "#ff6961"}
    options = {"ents": ['scope_neg', 'cue_neg'], "colors": colors}

    for i in range(len(sample)):
        entities = []
        for ents in sample[i]:
            entities.append({"end": ents["end"], "label": ents["entity_group"], "start": ents["start"]})
        displacy.render({"ents": entities,"text": text[i]}, style="ent", manual=True,options=options, jupyter=True)
visualize(sample,text)
```


#### Limitations and bias
The capacity of the Neg-CamemBERT-bio model is constrained due to the limited size of its training set, which includes a restricted number of examples for certain negation indicators 
*("sauf", "jamais", "hormis",...)* that appear less frequently. This limitation poses challenges for generalizing to other cases. 
The tokenizer, not specifically designed for radiology, can lead to confusion for the Neg-Radio-CamemBERT-bio model when predicting the scope. 
It is important to note that both the Radio corpus and the ESSAI + CAS corpus have been annotated by different annotators on purpose, 
which may introduce confusion into the ability of the Neg-Medical-CamemBERT-bio model to predict this scope accurately.