Initial commit

README.md

@@ -50,36 +50,6 @@ datasets:
 - mteb/sts12-sts
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
-metrics:
-- pearson_cosine
-- spearman_cosine
-- cosine_accuracy
-model-index:
-- name: SentenceTransformer
-  results:
-  - task:
-      type: semantic-similarity
-      name: Semantic Similarity
-    dataset:
-      name: Unknown
-      type: unknown
-    metrics:
-    - type: pearson_cosine
-      value: 0.2502604111969662
-      name: Pearson Cosine
-    - type: spearman_cosine
-      value: 0.2861642394156719
-      name: Spearman Cosine
-  - task:
-      type: triplet
-      name: Triplet
-    dataset:
-      name: Unknown
-      type: unknown
-    metrics:
-    - type: cosine_accuracy
-      value: 0.844
-      name: Cosine Accuracy
 ---
 
 # SentenceTransformer
@@ -170,27 +140,6 @@ You can finetune this model on your own dataset.
 *List how the model may foreseeably be misused and address what users ought not to do with the model.*
 -->
 
-## Evaluation
-
-### Metrics
-
-#### Semantic Similarity
-
-* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
-
-| Metric              | Value      |
-|:--------------------|:-----------|
-| pearson_cosine      | 0.2503     |
-| **spearman_cosine** | **0.2862** |
-
-#### Triplet
-
-* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
-
-| Metric              | Value     |
-|:--------------------|:----------|
-| **cosine_accuracy** | **0.844** |
-
 <!--
 ## Bias, Risks and Limitations
 
@@ -264,7 +213,7 @@ You can finetune this model on your own dataset.
 - `per_device_train_batch_size`: 32
 - `per_device_eval_batch_size`: 32
 - `learning_rate`: 1e-05
-- `num_train_epochs`: 10
+- `num_train_epochs`: 1
 
 #### All Hyperparameters
 <details><summary>Click to expand</summary>
@@ -286,7 +235,7 @@ You can finetune this model on your own dataset.
 - `adam_beta2`: 0.999
 - `adam_epsilon`: 1e-08
 - `max_grad_norm`: 1.0
-- `num_train_epochs`: 10
+- `num_train_epochs`: 1
 - `max_steps`: -1
 - `lr_scheduler_type`: linear
 - `lr_scheduler_kwargs`: {}
@@ -387,24 +336,13 @@ You can finetune this model on your own dataset.
 
 </details>
 
-### Training Logs
-| Epoch | Step | Training Loss | Validation Loss | spearman_cosine | cosine_accuracy |
-|:-----:|:----:|:-------------:|:---------------:|:---------------:|:---------------:|
-| 3.125 | 100  | 6.523         | 6.3663          | 0.2497          | -               |
-| 6.25  | 200  | 6.0248        | 6.3467          | 0.2702          | -               |
-| 9.375 | 300  | 5.8616        | 6.3936          | 0.2862          | -               |
-| 3.125 | 100  | 2.1251        | 1.2034          | -               | 0.854           |
-| 6.25  | 200  | 1.6618        | 1.2496          | -               | 0.843           |
-| 9.375 | 300  | 1.6239        | 1.2676          | -               | 0.844           |
-
-
 ### Framework Versions
 - Python: 3.10.12
 - Sentence Transformers: 3.3.1
 - Transformers: 4.46.2
 - PyTorch: 2.5.1+cu121
 - Accelerate: 1.1.1
-- Datasets: 3.1.0
+- Datasets: 2.21.0
 - Tokenizers: 0.20.3
 
 ## Citation

custom_trans.py

@@ -0,0 +1,46 @@
+import torch
+import torch.nn as nn
+from sentence_transformers import models
+
+class CustTrans(models.Transformer):
+
+  def __init__(self, *args, **kwargs):
+      super().__init__(*args, **kwargs)
+      self.curr_task_type = None
+      self._rebuild_taskembedding(['sts', 'quora'])
+
+  def forward(self, inputs, task_type=None):
+
+    enc = self.auto_model(**inputs).last_hidden_state
+
+    if task_type == None:
+      task_type = self.curr_task_type
+
+    if  task_type in self.task_types:
+      idx = torch.tensor(self.task_types.index(task_type), device=self.TaskEmbedding.weight.device)
+      hyp = self.TaskEmbedding(idx)
+      inputs['token_embeddings'] = self._project(enc, hyp)
+
+    else:
+      inputs['token_embeddings'] = enc
+
+    return inputs
+
+  def _set_curr_task_type(self, task_type):
+    self.curr_task_type = task_type
+
+  def _set_taskembedding_grad(self, value):
+        self.TaskEmbedding.weight.requires_grad = value
+
+  def _set_transformer_grad(self, value):
+        for param in self.auto_model.parameters():
+          param.requires_grad = value
+
+  def _rebuild_taskembedding(self, task_types):
+    self.task_types = task_types
+    self.task_emb = 1 - torch.eye(len(self.task_types),768)
+    self.TaskEmbedding = nn.Embedding(len(self.task_types), 768).from_pretrained(self.task_emb)
+
+  def _project(self, v, normal_hyper):
+    # return v - torch.dot(v, normal_hyper)*normal_hyper / torch.norm(normal_hyper)**2
+    return v*normal_hyper

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f3f5cb0e45d40583bb1696b6113195cb1a34650541ded587f9e9cdee78985248
+oid sha256:8e47716a979def3ee4331621abb95a2a07619cf6428ca798c051201cbbc0ff89
 size 437951328

modules.json

@@ -3,7 +3,7 @@
     "idx": 0,
     "name": "0",
     "path": "",
-    "type": "__main__.CustTrans"
+    "type": "custom_trans.CustTrans"
   },
   {
     "idx": 1,