File size: 6,074 Bytes
4c5ad14 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 4c5ad14 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d 567996f 83ef30d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
---
license: mit
datasets:
- ai4privacy/open-pii-masking-500k-ai4privacy
language:
- fr
- en
- de
- te
- hi
- it
- es
- nl
base_model:
- answerdotai/ModernBERT-base
library_name: transformers
tags:
- PII
- redaction
- anonymisation
- token-classification
model-index:
- name: multilingual-anonymiser-openpii-ai4privacy
results:
- task:
type: token-classification
name: PII Masking and Classification
dataset:
type: ai4privacy/open-pii-masking-500k-ai4privacy
name: Open PII Masking 500K
split: test
metrics:
- type: f1
value: 0.9150
name: F1 Score
- type: precision
value: 0.8761
name: Precision
- type: recall
value: 0.9576
name: Recall
- type: accuracy
value: 0.9503
name: Accuracy
---
# Multilingual Anonymiser OpenPII (Ai4Privacy)
This model is designed to **redact and classify Personally Identifiable Information (PII)** from multilingual text. It has been fine-tuned on the [open-pii-masking-500k-ai4privacy](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy) dataset and supports multiple languages, including French (fr), English (en), German (de), Telugu (te), Hindi (hi), Italian (it), Spanish (es), and Dutch (nl).
---
## Evaluation Metrics
The table below summarizes the detailed evaluation results per PII label. Metrics are presented as percentages rounded to two decimal places. For the "O" (Non-PII) label, precision, recall, and F1 score are not applicable (n/a) due to the absence of true positives.
| **Label** | **TP** | **FP** | **FN** | **Accuracy** | **Precision** | **Recall** | **F1 Score** |
|--------------------|:------:|:------:|:------|:------------:|:-------------:|:----------:|:------------:|
| O (Non-PII) | 0 | 734 | 0 | 98.97% | n/a | n/a | n/a |
| GIVENNAME | 6623 | 661 | 352 | 86.73% | 90.93% | 94.95% | 92.90% |
| SURNAME | 2786 | 877 | 162 | 72.84% | 76.06% | 94.50% | 84.28% |
| CITY | 1763 | 216 | 225 | 79.99% | 89.09% | 88.68% | 88.88% |
| DATE | 2195 | 1 | 3 | 99.82% | 99.95% | 99.86% | 99.91% |
| AGE | 176 | 7 | 2 | 95.14% | 96.17% | 98.88% | 97.51% |
| EMAIL | 2981 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% |
| CREDITCARDNUMBER | 601 | 57 | 35 | 86.72% | 91.34% | 94.50% | 92.89% |
| SEX | 103 | 45 | 1 | 69.13% | 69.59% | 99.04% | 81.75% |
| SOCIALNUM | 364 | 134 | 20 | 70.27% | 73.09% | 94.79% | 82.54% |
| TIME | 1631 | 1 | 3 | 99.76% | 99.94% | 99.82% | 99.88% |
| TELEPHONENUM | 3537 | 10 | 9 | 99.47% | 99.72% | 99.75% | 99.73% |
| IDCARDNUM | 1540 | 314 | 148 | 76.92% | 83.06% | 91.23% | 86.96% |
| ZIPCODE | 311 | 39 | 16 | 84.97% | 88.86% | 95.11% | 91.87% |
| DRIVERLICENSENUM | 296 | 143 | 26 | 63.66% | 67.43% | 91.93% | 77.79% |
| PASSPORTNUM | 482 | 285 | 25 | 60.86% | 62.84% | 95.07% | 75.67% |
| TITLE | 224 | 68 | 78 | 60.54% | 76.71% | 74.17% | 75.42% |
| BUILDINGNUM | 292 | 45 | 14 | 83.19% | 86.65% | 95.42% | 90.85% |
| STREET | 1272 | 155 | 67 | 85.14% | 89.14% | 94.99% | 91.97% |
| TAXNUM | 471 | 101 | 34 | 77.72% | 82.34% | 93.27% | 87.47% |
| GENDER | 123 | 35 | 9 | 73.65% | 77.85% | 93.18% | 84.83% |
### Overall Evaluation
- **Accuracy:** 95.03%
- **Precision:** 87.61%
- **Recall:** 95.76%
- **F1 Score:** 91.50%
- **Total True Positives (TP):** 27,771
- **Total False Positives (FP):** 3,928
- **Total False Negatives (FN):** 1,229
### Macro-Averaged Metrics
- **Accuracy:** 82.17%
- **Precision:** 80.99%
- **Recall:** 89.96%
- **F1 Score:** 84.91%
---
## Model Behavior & Limitations
- **Evaluation Focus:**
The metrics above reflect performance on the test split of the [open-pii-masking-500k-ai4privacy](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy) dataset. This model both redacts and classifies PII into specific categories (e.g., GIVENNAME, EMAIL). Real-world performance may vary depending on text domain and language, so additional validation is recommended. For support, contact **[email protected]**.
- **Strengths:**
- High recall (95.76%) ensures most PII is detected.
- Exceptional performance on labels like "EMAIL" (100% F1), "DATE" (99.91% F1), and "TIME" (99.88% F1).
- **Limitations:**
- Lower precision for labels such as "PASSPORTNUM" (62.84%) and "DRIVERLICENSENUM" (67.43%), indicating a higher rate of false positives.
- The "O" (Non-PII) label has no true positives, making precision, recall, and F1 score not applicable (n/a).
---
## Disclaimer
This model card details the evaluation metrics and fine-tuning parameters for the multilingual anonymiser with PII classification capabilities. **Please note:**
- The model is provided **as-is** under the MIT License.
- It is intended for both redaction and PII classification purposes.
- Users should thoroughly test and evaluate its performance on their specific datasets before deploying in production environments.
---
*Ai4Privacy – Committed to protecting personal data in the age of AI.*
--- |