---
license: apache-2.0
datasets:
- code-search-net/code_search_net
language:
- en
pipeline_tag: fill-mask
tags:
- code
- python
- java
- javascript
- go
- ruby
- php
---

# CodeModernBERT-Finch

This model is a code-specific pretrained model created solely using the CodeSearchNet dataset. It supports six languages included in CodeSearchNet.\
For a version fine-tuned specifically for code search tasks, please refer to [Shuu12121/CodeSearch-ModernBERT-Finch](https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Finch).

## Architecture

* Base: ModernBERT-style encoder
* Hidden size: 512
* Layers: 6
* Attention heads: 6
* Parameters: \~50M
* Pretraining: Masked Language Modeling (MLM)
* Fine-tuning: Domain-specific code tasks

The results below were obtained by randomly sampling 10,000 examples per language from the CodeSearchNet dataset, training them in a Sentence-BERT fashion, and evaluating on the MTEB CodeSearchNetRetrieval benchmark.
All models listed in the table below were fine-tuned using the same approach. Those marked with 200 and the Finch models were trained with a Multiple Negatives Ranking Loss batch size of 200. Others were trained with a batch size of 40 (because larger batches could not fit into memory).\
Finch-SmallBatch was trained with a smaller batch size of 40 to create a comparison model against the standard Finch models trained with batch size 200.

| Model                              | go    | java  | javascript | php   | python | ruby  |
| ---------------------------------- | ----- | ----- | ---------- | ----- | ------ | ----- |
| Finch(40M)                         | 0.934 | 0.784 | 0.728      | 0.835 | 0.865  | 0.756 |
| Finch-Pre(40M)                     | 0.937 | 0.705 | 0.685      | 0.828 | 0.843  | 0.725 |
| Finch-SmallBatch(40M)             | 0.930 | 0.765 | 0.707     | 0.825  | 0.859 | 0.748 |
| ModernBERT-base-Finetuned(149M)               | 0.933 | 0.779 | 0.748      | 0.839 | 0.885  | 0.794 |
| Owl-4.1-Small-Fine-tuned(151M)     | 0.942 | 0.780 | 0.729      | 0.843 | 0.893  | 0.772 |
| Owl-4.1-Small-Fine-tuned-200(151M) | 0.943 | 0.850 | 0.747      | 0.858 | 0.894  | 0.802 |
| CodeBERT-Fine-tuned(125M)          | 0.932 | 0.708 | 0.709      | 0.828 | 0.870  | 0.772 |


---