readme: ---
  pipeline_tag: sentence-similarity
  tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - mteb
  model-index:
  - name: sgpt-bloom-1b7-nli
    results:
    - task:
        type: Classification
      dataset:
        type: mteb/amazon_reviews_multi
        name: MTEB AmazonReviewsClassification (fr)
        config: fr
        split: test
        revision: c379a6705fec24a2493fa68e011692605f44e119
      metrics:
      - type: accuracy
        value: 39.286
      - type: f1
        value: 38.87078070073539
    - task:
        type: Classification
      dataset:
        type: mteb/amazon_reviews_multi
        name: MTEB AmazonReviewsClassification (zh)
        config: zh
        split: test
        revision: c379a6705fec24a2493fa68e011692605f44e119
      metrics:
      - type: accuracy
        value: 37.634
      - type: f1
        value: 36.86046604093418
    - task:
        type: Classification
      dataset:
        type: mteb/mtop_domain
        name: MTEB MTOPDomainClassification (fr)
        config: fr
        split: test
        revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
      metrics:
      - type: accuracy
        value: 83.79893517068588
      - type: f1
        value: 83.72326662566203
    - task:
        type: Classification
      dataset:
        type: mteb/mtop_intent
        name: MTEB MTOPIntentClassification (fr)
        config: fr
        split: test
        revision: 6299947a7777084cc2d4b64235bf7190381ce755
      metrics:
      - type: accuracy
        value: 63.36047604134043
      - type: f1
        value: 44.261707019308126
    - task:
        type: Classification
      dataset:
        type: mteb/amazon_massive_intent
        name: MTEB MassiveIntentClassification (fr)
        config: fr
        split: test
        revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
      metrics:
      - type: accuracy
        value: 64.57632817753867
      - type: f1
        value: 62.60453982786661
    - task:
        type: Classification
      dataset:
        type: mteb/amazon_massive_scenario
        name: MTEB MassiveScenarioClassification (fr)
        config: fr
        split: test
        revision: 7d571f92784cd94a019292a1f45445077d0ef634
      metrics:
      - type: accuracy
        value: 69.59986550100874
      - type: f1
        value: 69.71803697939914
    - task:
        type: STS
      dataset:
        type: mteb/sts22-crosslingual-sts
        name: MTEB STS22 (zh)
        config: zh
        split: test
        revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
      metrics:
      - type: cos_sim_pearson
        value: 59.71781185663265
      - type: cos_sim_spearman
        value: 58.538648447630514
      - type: euclidean_pearson
        value: 53.53848180206165
      - type: euclidean_spearman
        value: 56.33730262964236
      - type: manhattan_pearson
        value: 54.62109820575505
      - type: manhattan_spearman
        value: 57.223846291318914
    - task:
        type: STS
      dataset:
        type: mteb/sts22-crosslingual-sts
        name: MTEB STS22 (fr)
        config: fr
        split: test
        revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
      metrics:
      - type: cos_sim_pearson
        value: 73.44021434651606
      - type: cos_sim_spearman
        value: 73.13412769502769
      - type: euclidean_pearson
        value: 68.16368597409867
      - type: euclidean_spearman
        value: 72.44964781564485
      - type: manhattan_pearson
        value: 69.42307032478939
      - type: manhattan_spearman
        value: 73.3523195012387
  ---
  
  # sgpt-bloom-1b7-nli
  
  ## Usage
  
  For usage instructions, refer to: https://github.com/Muennighoff/sgpt#symmetric-semantic-search
  
  The model was trained with the command
  ```bash
  CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/nli/training_nli_v2.py --model_name bigscience/bloom-1b3 --freezenonbias --train_batch_size 128 --lr 32e-5 --pooling weightedmean --wandb --wandbwatchlog gradients --gradcache --chunksize 4
  ```
  
  ## Evaluation Results
  
  `{'askubuntu': 57.44, 'cqadupstack': 14.18, 'twitterpara': 73.99, 'scidocs': 74.74, 'avg': 55.087500000000006}`
  
  ## Training
  The model was trained with the parameters:
  
  **DataLoader**:
  
  `sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader` of length 4403 with parameters:
  ```
  {'batch_size': 128}
  ```
  
  The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904
  
  **Loss**:
  
  `sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache` 
  
  Parameters of the fit()-Method:
  ```
  {
      "epochs": 1,
      "evaluation_steps": 440,
      "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
      "max_grad_norm": 1,
      "optimizer_class": "<class 'transformers.optimization.AdamW'>",
      "optimizer_params": {
          "lr": 0.00032
      },
      "scheduler": "WarmupLinear",
      "steps_per_epoch": null,
      "warmup_steps": 441,
      "weight_decay": 0.01
  }
  ```
  
  
  ## Full Model Architecture
  ```
  SentenceTransformer(
    (0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: BloomModel 
    (1): Pooling({'word_embedding_dimension': 2048, 'pooling