Update README.md
Browse files
README.md
CHANGED
|
@@ -69,7 +69,16 @@ model = get_peft_model(model, peft_config)
|
|
| 69 |
|
| 70 |
### Intermediate Checkpoints
|
| 71 |
In addition to the final model checkpoint, we publish intermediate checkpoints throughout the full training process as unique branches in this repository.
|
|
|
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
### Performance
|
| 75 |
We evaluate our models across a broad range of tasks. For natural language understanding, we use the [SuperGLEBer](https://lsx-uniwue.github.io/SuperGLEBer-site/) benchmark, and for embedding capabilities, we use the [German MTEB](http://mteb-leaderboard.hf.space/?benchmark_name=MTEB%28deu%2C+v1%29) benchmark (after unsupervised fine-tuning of every model on the German mMARCO portion). The following table provides a comparison of this encoder with other German and multilingual encoders. See our [preprint](https://arxiv.org/abs/2505.13136) for more details about the evaluation.
|
|
|
|
| 69 |
|
| 70 |
### Intermediate Checkpoints
|
| 71 |
In addition to the final model checkpoint, we publish intermediate checkpoints throughout the full training process as unique branches in this repository.
|
| 72 |
+
A specific checkpoint can be loaded like this:
|
| 73 |
|
| 74 |
+
```python
|
| 75 |
+
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
| 76 |
+
|
| 77 |
+
model_id = "LSX-UniWue/ModernGBERT_1B"
|
| 78 |
+
revision = "base-head-12000-ckpt"
|
| 79 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
|
| 80 |
+
model = AutoModelForMaskedLM.from_pretrained(model_id, revision=revision)
|
| 81 |
+
```
|
| 82 |
|
| 83 |
### Performance
|
| 84 |
We evaluate our models across a broad range of tasks. For natural language understanding, we use the [SuperGLEBer](https://lsx-uniwue.github.io/SuperGLEBer-site/) benchmark, and for embedding capabilities, we use the [German MTEB](http://mteb-leaderboard.hf.space/?benchmark_name=MTEB%28deu%2C+v1%29) benchmark (after unsupervised fine-tuning of every model on the German mMARCO portion). The following table provides a comparison of this encoder with other German and multilingual encoders. See our [preprint](https://arxiv.org/abs/2505.13136) for more details about the evaluation.
|