End of training

README.md

@@ -0,0 +1,87 @@
+---
+base_model: AIRI-Institute/gena-lm-bert-large-t2t
+tags:
+- generated_from_trainer
+metrics:
+- precision
+- recall
+- accuracy
+model-index:
+- name: gena-lm-bert-large-t2t_ft_BioS2_1kbpHG19_DHSs_H3K27AC
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# gena-lm-bert-large-t2t_ft_BioS2_1kbpHG19_DHSs_H3K27AC
+
+This model is a fine-tuned version of [AIRI-Institute/gena-lm-bert-large-t2t](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.3605
+- F1 Score: 0.8722
+- Precision: 0.8099
+- Recall: 0.9449
+- Accuracy: 0.8549
+- Auc: 0.9416
+- Prc: 0.9432
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 20
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss | F1 Score | Precision | Recall | Accuracy | Auc    | Prc    |
+|:-------------:|:------:|:-----:|:---------------:|:--------:|:---------:|:------:|:--------:|:------:|:------:|
+| 0.5815        | 0.0840 | 500   | 0.5396          | 0.7416   | 0.8093    | 0.6843 | 0.7500   | 0.8334 | 0.8249 |
+| 0.4864        | 0.1680 | 1000  | 0.4740          | 0.8142   | 0.7803    | 0.8513 | 0.7964   | 0.8687 | 0.8579 |
+| 0.4701        | 0.2520 | 1500  | 0.5119          | 0.7828   | 0.8415    | 0.7317 | 0.7872   | 0.8688 | 0.8706 |
+| 0.4654        | 0.3360 | 2000  | 0.4932          | 0.8274   | 0.7359    | 0.9449 | 0.7934   | 0.8922 | 0.8869 |
+| 0.4376        | 0.4200 | 2500  | 0.4289          | 0.8365   | 0.8007    | 0.8756 | 0.8206   | 0.8984 | 0.8962 |
+| 0.423         | 0.5039 | 3000  | 0.4881          | 0.8366   | 0.7595    | 0.9311 | 0.8093   | 0.9056 | 0.9034 |
+| 0.4226        | 0.5879 | 3500  | 0.3795          | 0.8504   | 0.8110    | 0.8939 | 0.8352   | 0.9131 | 0.9133 |
+| 0.4035        | 0.6719 | 4000  | 0.3801          | 0.8491   | 0.7886    | 0.9196 | 0.8287   | 0.9186 | 0.9217 |
+| 0.4041        | 0.7559 | 4500  | 0.3687          | 0.8553   | 0.8172    | 0.8971 | 0.8409   | 0.9226 | 0.9244 |
+| 0.3948        | 0.8399 | 5000  | 0.4052          | 0.8421   | 0.7465    | 0.9657 | 0.8102   | 0.9290 | 0.9314 |
+| 0.3914        | 0.9239 | 5500  | 0.4325          | 0.8483   | 0.7639    | 0.9538 | 0.8213   | 0.9294 | 0.9319 |
+| 0.3801        | 1.0079 | 6000  | 0.3344          | 0.8677   | 0.8623    | 0.8731 | 0.8604   | 0.9339 | 0.9368 |
+| 0.3334        | 1.0919 | 6500  | 0.3710          | 0.8695   | 0.8301    | 0.9128 | 0.8564   | 0.9337 | 0.9368 |
+| 0.3567        | 1.1759 | 7000  | 0.3799          | 0.8672   | 0.8740    | 0.8606 | 0.8619   | 0.9372 | 0.9398 |
+| 0.3528        | 1.2599 | 7500  | 0.3344          | 0.8698   | 0.8609    | 0.8788 | 0.8621   | 0.9378 | 0.9404 |
+| 0.3542        | 1.3439 | 8000  | 0.3936          | 0.8769   | 0.8249    | 0.9359 | 0.8623   | 0.9393 | 0.9405 |
+| 0.3497        | 1.4279 | 8500  | 0.3461          | 0.8730   | 0.8586    | 0.8878 | 0.8646   | 0.9393 | 0.9403 |
+| 0.3379        | 1.5118 | 9000  | 0.4219          | 0.8683   | 0.7987    | 0.9513 | 0.8488   | 0.9408 | 0.9417 |
+| 0.347         | 1.5958 | 9500  | 0.3649          | 0.8746   | 0.8491    | 0.9016 | 0.8644   | 0.9391 | 0.9409 |
+| 0.3513        | 1.6798 | 10000 | 0.3315          | 0.8655   | 0.8944    | 0.8385 | 0.8634   | 0.9414 | 0.9438 |
+| 0.3259        | 1.7638 | 10500 | 0.3605          | 0.8722   | 0.8099    | 0.9449 | 0.8549   | 0.9416 | 0.9432 |
+
+
+### Framework versions
+
+- Transformers 4.42.3
+- Pytorch 2.3.0+cu121
+- Datasets 2.18.0
+- Tokenizers 0.19.0

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "AIRI-Institute/gena-lm-bert-large-t2t",
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoModel": "AIRI-Institute/gena-lm-bert-large-t2t--modeling_bert.BertForMaskedLM"
+  },
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 3,
+  "position_embedding_type": "absolute",
+  "pre_layer_norm": true,
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89dd47a346e072a92849869e13b6e2c05263c9dbc825f95473cdc1692c8f2256
+size 1346688816

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,60 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "-",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2091efbbfb0f6772794ddbef0729440f3c9fb3153517de42de3ad353353c45c
+size 5240