transformers t5 feature-extraction ctranslate2 int8 float16 - finetuner - mteb - sentence-transformers - feature-extraction - sentence-similarity sentence-similarity custom_code en dataset:jinaai/negation-dataset arxiv:2307.11224 license:apache-2.0 model-index region:us
---
datasets:
- jinaai/negation-dataset
language: en
license: apache-2.0
pipeline_tag: sentence-similarity
tags:
- ctranslate2
- int8
- float16 - finetuner - mteb - sentence-transformers - feature-extraction - sentence-similarity
model-index:
- name: jina-embedding-s-en-v1
  results:
  - task:
      type: Classification
    dataset:
      name: MTEB AmazonCounterfactualClassification (en)
      type: mteb/amazon_counterfactual
      config: en
      split: test
      revision: e8379541af4e31359cca9fbcf4b00f2671dba205
    metrics:
    - type: accuracy
      value: 64.82089552238806
    - type: ap
      value: 27.100981946230778
    - type: f1
      value: 58.3354886367184
  - task:
      type: Classification
    dataset:
      name: MTEB AmazonPolarityClassification
      type: mteb/amazon_polarity
      config: default
      split: test
      revision: e2d317d38cd51312af73b3d32a06d1a08b442046
    metrics:
    - type: accuracy
      value: 64.282775
    - type: ap
      value: 60.350688924943796
    - type: f1
      value: 62.06346948494396
  - task:
      type: Classification
    dataset:
      name: MTEB AmazonReviewsClassification (en)
      type: mteb/amazon_reviews_multi
      config: en
      split: test
      revision: 1399c76144fd37290681b995c656ef9b2e06e26d
    metrics:
    - type: accuracy
      value: 30.623999999999995
    - type: f1
      value: 29.427789186742153
  - task:
      type: Retrieval
    dataset:
      name: MTEB ArguAna
      type: arguana
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 22.119
    - type: map_at_10
      value: 35.609
    - type: map_at_100
      value: 36.935
    - type: map_at_1000
      value: 36.957
    - type: map_at_3
      value: 31.046000000000003
    - type: map_at_5
      value: 33.574
    - type: mrr_at_1
      value: 22.404
    - type: mrr_at_10
      value: 35.695
    - type: mrr_at_100
      value: 37.021
    - type: mrr_at_1000
      value: 37.043
    - type: mrr_at_3
      value: 31.093
    - type: mrr_at_5
      value: 33.635999999999996
    - type: ndcg_at_1
      value: 22.119
    - type: ndcg_at_10
      value: 43.566
    - type: ndcg_at_100
      value: 49.370000000000005
    - type: ndcg_at_1000
      value: 49.901
    - type: ndcg_at_3
      value: 34.06
    - type: ndcg_at_5
      value: 38.653999999999996
    - type: precision_at_1
      value: 22.119
    - type: precision_at_10
      value: 6.92
    - type: precision_at_100
      value: 0.95
    - type: precision_at_1000
      value: 0.099
    - type: precision_at_3
      value: 14.272000000000002
    - type: precision_at_5
      value: 10.811
    - type: recall_at_1
      value: 22.119
    - type: recall_at_10
      value: 69.203
    - type: recall_at_100
      value: 95.021
    - type: recall_at_1000
      value: 99.075
    - type: recall_at_3
      value: 42.817
    - type: recall_at_5
      value: 54.054
  - task:
      type: Clustering
    dataset:
      name: MTEB ArxivClusteringP2P
      type: mteb/arxiv-clustering-p2p
      config: default
      split: test
      revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
    metrics:
    - type: v_measure
      value: 34.1740289109719
  - task:
      type: Clustering
    dataset:
      name: MTEB ArxivClusteringS2S
      type: mteb/arxiv-clustering-s2s
      config: default
      split: test
      revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
    metrics:
    - type: v_measure
      value: 23.985251383455463
  - task:
      type: Reranking
    dataset:
      name: MTEB AskUbuntuDupQuestions
      type: mteb/askubuntudupquestions-reranking
      config: default
      split: test
      revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
    metrics:
    - type: map
      value: 60.24873612289029
    - type: mrr
      value: 74.65692740623489
  - task:
      type: STS
    dataset:
      name: MTEB BIOSSES
      type: mteb/biosses-sts
      config: default
      split: test
      revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
    metrics:
    - type: cos_sim_pearson
      value: 86.22415390332444
    - type: cos_sim_spearman
      value: 82.9591191954711
    - type: euclidean_pearson
      value: 44.096317524324945
    - type: euclidean_spearman
      value: 42.95218351391625
    - type: manhattan_pearson
      value: 44.07766490545065
    - type: manhattan_spearman
      value: 42.78350497166606
  - task:
      type: Classification
    dataset:
      name: MTEB Banking77Classification
      type: mteb/banking77
      config: default
      split: test
      revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
    metrics:
    - type: accuracy
      value: 74.64285714285714
    - type: f1
      value: 73.53680835577447
  - task:
      type: Clustering
    dataset:
      name: MTEB BiorxivClusteringP2P
      type: mteb/biorxiv-clustering-p2p
      config: default
      split: test
      revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
    metrics:
    - type: v_measure
      value: 28.512813238490164
  - task:
      type: Clustering
    dataset:
      name: MTEB BiorxivClusteringS2S
      type: mteb/biorxiv-clustering-s2s
      config: default
      split: test
      revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
    metrics:
    - type: v_measure
      value: 20.942214972649488
  - task:
      type: Retrieval
    dataset:
      name: MTEB CQADupstackAndroidRetrieval
      type: BeIR/cqadupstack
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 28.255999999999997
    - type: map_at_10
      value: 37.091
    - type: map_at_100
      value: 38.428000000000004
    - type: map_at_1000
      value: 38.559
    - type: map_at_3
      value: 34.073
    - type: map_at_5
      value: 35.739
    - type: mrr_at_1
      value: 34.907
    - type: mrr_at_10
      value: 42.769
    - type: mrr_at_100
      value: 43.607
    - type: mrr_at_1000
      value: 43.656
    - type: mrr_at_3
      value: 39.986
    - type: mrr_at_5
      value: 41.581
    - type: ndcg_at_1
      value: 34.907
    - type: ndcg_at_10
      value: 42.681000000000004
    - type: ndcg_at_100
      value: 48.213
    - type: ndcg_at_1000
      value: 50.464
    - type: ndcg_at_3
      value: 37.813
    - type: ndcg_at_5
      value: 39.936
    - type: precision_at_1
      value: 34.907
    - type: precision_at_10
      value: 7.911
    - type: precision_at_100
      value: 1.349
    - type: precision_at_1000
      value: 0.184
    - type: precision_at_3
      value: 17.93
    - type: precision_at_5
      value: 12.732
    - type: recall_at_1
      value: 28.255999999999997
    - type: recall_at_10
      value: 53.49699999999999
    - type: recall_at_100
      value: 77.288
    - type: recall_at_1000
      value: 91.776
    - type: recall_at_3
      value: 39.18
    - type: recall_at_5
      value: 45.365
    - type: map_at_1
      value: 25.563999999999997
    - type: map_at_10
      value: 33.913
    - type: map_at_100
      value: 34.966
    - type: map_at_1000
      value: 35.104
    - type: map_at_3
      value: 31.413000000000004
    - type: map_at_5
      value: 32.854
    - type: mrr_at_1
      value: 31.72
    - type: mrr_at_10
      value: 39.391
    - type: mrr_at_100
      value: 40.02
    - type: mrr_at_1000
      value: 40.076
    - type: mrr_at_3
      value: 37.314
    - type: mrr_at_5
      value: 38.507999999999996
    - type: ndcg_at_1
      value: 31.72
    - type: ndcg_at_10
      value: 38.933
    - type: ndcg_at_100
      value: 43.024
    - type: ndcg_at_1000
      value: 45.556999999999995
    - type: ndcg_at_3
      value: 35.225
    - type: ndcg_at_5
      value: 36.984
    - type: precision_at_1
      value: 31.72
    - type: precision_at_10
      value: 7.248
    - type: precision_at_100
      value: 1.192
    - type: precision_at_1000
      value: 0.16999999999999998
    - type: precision_at_3
      value: 16.943
    - type: precision_at_5
      value: 11.975
    - type: recall_at_1
      value: 25.563999999999997
    - type: recall_at_10
      value: 47.808
    - type: recall_at_100
      value: 65.182
    - type: recall_at_1000
      value: 81.831
    - type: recall_at_3
      value: 36.889
    - type: recall_at_5
      value: 41.829
    - type: map_at_1
      value: 33.662
    - type: map_at_10
      value: 44.096999999999994
    - type: map_at_100
      value: 45.153999999999996
    - type: map_at_1000
      value: 45.223
    - type: map_at_3
      value: 41.377
    - type: map_at_5
      value: 42.935
    - type: mrr_at_1
      value: 38.997
    - type: mrr_at_10
      value: 47.675
    - type: mrr_at_100
      value: 48.476
    - type: mrr_at_1000
      value: 48.519
    - type: mrr_at_3
      value: 45.549
    - type: mrr_at_5
      value: 46.884
    - type: ndcg_at_1
      value: 38.997
    - type: ndcg_at_10
      value: 49.196
    - type: ndcg_at_100
      value: 53.788000000000004
    - type: ndcg_at_1000
      value: 55.393
    - type: ndcg_at_3
      value: 44.67
    - type: ndcg_at_5
      value: 46.991
    - type: precision_at_1
      value: 38.997
    - type: precision_at_10
      value: 7.875
    - type: precision_at_100
      value: 1.102
    - type: precision_at_1000
      value: 0.13
    - type: precision_at_3
      value: 19.854
    - type: precision_at_5
      value: 13.605
    - type: recall_at_1
      value: 33.662
    - type: recall_at_10
      value: 60.75899999999999
    - type: recall_at_100
      value: 81.11699999999999
    - type: recall_at_1000
      value: 92.805
    - type: recall_at_3
      value: 48.577999999999996
    - type: recall_at_5
      value: 54.384
    - type: map_at_1
      value: 21.313
    - type: map_at_10
      value: 29.036
    - type: map_at_100
      value: 29.975
    - type: map_at_1000
      value: 30.063000000000002
    - type: map_at_3
      value: 26.878999999999998
    - type: map_at_5
      value: 28.005999999999997
    - type: mrr_at_1
      value: 23.39
    - type: mrr_at_10
      value: 31.072
    - type: mrr_at_100
      value: 31.922
    - type: mrr_at_1000
      value: 31.995
    - type: mrr_at_3
      value: 28.908
    - type: mrr_at_5
      value: 30.104999999999997
    - type: ndcg_at_1
      value: 23.39
    - type: ndcg_at_10
      value: 33.448
    - type: ndcg_at_100
      value: 38.255
    - type: ndcg_at_1000
      value: 40.542
    - type: ndcg_at_3
      value: 29.060000000000002
    - type: ndcg_at_5
      value: 31.023
    - type: precision_at_1
      value: 23.39
    - type: precision_at_10
      value: 5.175
    - type: precision_at_100
      value: 0.8049999999999999
    - type: precision_at_1000
      value: 0.10300000000000001
    - type: precision_at_3
      value: 12.504999999999999
    - type: precision_at_5
      value: 8.61
    - type: recall_at_1
      value: 21.313
    - type: recall_at_10
      value: 45.345
    - type: recall_at_100
      value: 67.752
    - type: recall_at_1000
      value: 84.937
    - type: recall_at_3
      value: 33.033
    - type: recall_at_5
      value: 37.929
    - type: map_at_1
      value: 14.255999999999998
    - type: map_at_10
      value: 20.339
    - type: map_at_100
      value: 21.491
    - type: map_at_1000
      value: 21.616
    - type: map_at_3
      value: 18.481
    - type: map_at_5
      value: 19.594
    - type: mrr_at_1
      value: 17.413
    - type: mrr_at_10
      value: 24.146
    - type: mrr_at_100
      value: 25.188
    - type: mrr_at_1000
      value: 25.273
    - type: mrr_at_3
      value: 22.264
    - type: mrr_at_5
      value: 23.302
    - type: ndcg_at_1
      value: 17.413
    - type: ndcg_at_10
      value: 24.272
    - type: ndcg_at_100
      value: 29.82
    - type: ndcg_at_1000
      value: 33.072
    - type: ndcg_at_3
      value: 20.826
    - type: ndcg_at_5
      value: 22.535
    - type: precision_at_1
      value: 17.413
    - type: precision_at_10
      value: 4.366
    - type: precision_at_100
      value: 0.818
    - type: precision_at_1000
      value: 0.124
    - type: precision_at_3
      value: 9.866999999999999
    - type: precision_at_5
      value: 7.164
    - type: recall_at_1
      value: 14.255999999999998
    - type: recall_at_10
      value: 32.497
    - type: recall_at_100
      value: 56.592
    - type: recall_at_1000
      value: 80.17699999999999
    - type: recall_at_3
      value: 23.195
    - type: recall_at_5
      value: 27.392
    - type: map_at_1
      value: 22.709
    - type: map_at_10
      value: 31.377
    - type: map_at_100
      value: 32.536
    - type: map_at_1000
      value: 32.669
    - type: map_at_3
      value: 28.572999999999997
    - type: map_at_5
      value: 30.205
    - type: mrr_at_1
      value: 27.815
    - type: mrr_at_10
      value: 36.452
    - type: mrr_at_100
      value: 37.302
    - type: mrr_at_1000
      value: 37.364000000000004
    - type: mrr_at_3
      value: 33.75
    - type: mrr_at_5
      value: 35.43
    - type: ndcg_at_1
      value: 27.815
    - type: ndcg_at_10
      value: 36.84
    - type: ndcg_at_100
      value: 42.092
    - type: ndcg_at_1000
      value: 44.727
    - type: ndcg_at_3
      value: 31.964
    - type: ndcg_at_5
      value: 34.428
    - type: precision_at_1
      value: 27.815
    - type: precision_at_10
      value: 6.67
    - type: precision_at_100
      value: 1.093
    - type: precision_at_1000
      value: 0.151
    - type: precision_at_3
      value: 14.982000000000001
    - type: precision_at_5
      value: 10.857
    - type: recall_at_1
      value: 22.709
    - type: recall_at_10
      value: 48.308
    - type: recall_at_100
      value: 70.866
    - type: recall_at_1000
      value: 88.236
    - type: recall_at_3
      value: 34.709
    - type: recall_at_5
      value: 40.996
    - type: map_at_1
      value: 22.348000000000003
    - type: map_at_10
      value: 29.427999999999997
    - type: map_at_100
      value: 30.499
    - type: map_at_1000
      value: 30.631999999999998
    - type: map_at_3
      value: 27.035999999999998
    - type: map_at_5
      value: 28.351
    - type: mrr_at_1
      value: 27.74
    - type: mrr_at_10
      value: 34.424
    - type: mrr_at_100
      value: 35.341
    - type: mrr_at_1000
      value: 35.419
    - type: mrr_at_3
      value: 32.401
    - type: mrr_at_5
      value: 33.497
    - type: ndcg_at_1
      value: 27.74
    - type: ndcg_at_10
      value: 34.136
    - type: ndcg_at_100
      value: 39.269
    - type: ndcg_at_1000
      value: 42.263
    - type: ndcg_at_3
      value: 30.171999999999997
    - type: ndcg_at_5
      value: 31.956
    - type: precision_at_1
      value: 27.74
    - type: precision_at_10
      value: 6.062
    - type: precision_at_100
      value: 1.014
    - type: precision_at_1000
      value: 0.146
    - type: precision_at_3
      value: 14.079
    - type: precision_at_5
      value: 9.977
    - type: recall_at_1
      value: 22.348000000000003
    - type: recall_at_10
      value: 43.477
    - type: recall_at_100
      value: 65.945
    - type: recall_at_1000
      value: 86.587
    - type: recall_at_3
      value: 32.107
    - type: recall_at_5
      value: 36.974000000000004
    - type: map_at_1
      value: 21.688499999999998
    - type: map_at_10
      value: 29.164666666666665
    - type: map_at_100
      value: 30.22575
    - type: map_at_1000
      value: 30.350833333333334
    - type: map_at_3
      value: 26.82025
    - type: map_at_5
      value: 28.14966666666667
    - type: mrr_at_1
      value: 25.779249999999998
    - type: mrr_at_10
      value: 32.969
    - type: mrr_at_100
      value: 33.81725
    - type: mrr_at_1000
      value: 33.88825
    - type: mrr_at_3
      value: 30.831250000000004
    - type: mrr_at_5
      value: 32.065000000000005
    - type: ndcg_at_1
      value: 25.779249999999998
    - type: ndcg_at_10
      value: 33.73675
    - type: ndcg_at_100
      value: 38.635666666666665
    - type: ndcg_at_1000
      value: 41.353500000000004
    - type: ndcg_at_3
      value: 29.66283333333333
    - type: ndcg_at_5
      value: 31.607249999999997
    - type: precision_at_1
      value: 25.779249999999998
    - type: precision_at_10
      value: 5.861416666666667
    - type: precision_at_100
      value: 0.9852500000000002
    - type: precision_at_1000
      value: 0.14108333333333334
    - type: precision_at_3
      value: 13.563583333333332
    - type: precision_at_5
      value: 9.630333333333335
    - type: recall_at_1
      value: 21.688499999999998
    - type: recall_at_10
      value: 43.605
    - type: recall_at_100
      value: 65.52366666666667
    - type: recall_at_1000
      value: 84.69683333333332
    - type: recall_at_3
      value: 32.195499999999996
    - type: recall_at_5
      value: 37.25325
    - type: map_at_1
      value: 17.279
    - type: map_at_10
      value: 23.238
    - type: map_at_100
      value: 24.026
    - type: map_at_1000
      value: 24.13
    - type: map_at_3
      value: 20.730999999999998
    - type: map_at_5
      value: 22.278000000000002
    - type: mrr_at_1
      value: 19.017999999999997
    - type: mrr_at_10
      value: 25.188
    - type: mrr_at_100
      value: 25.918999999999997
    - type: mrr_at_1000
      value: 25.996999999999996
    - type: mrr_at_3
      value: 22.776
    - type: mrr_at_5
      value: 24.256
    - type: ndcg_at_1
      value: 19.017999999999997
    - type: ndcg_at_10
      value: 27.171
    - type: ndcg_at_100
      value: 31.274
    - type: ndcg_at_1000
      value: 34.016000000000005
    - type: ndcg_at_3
      value: 22.442
    - type: ndcg_at_5
      value: 24.955
    - type: precision_at_1
      value: 19.017999999999997
    - type: precision_at_10
      value: 4.494
    - type: precision_at_100
      value: 0.712
    - type: precision_at_1000
      value: 0.10300000000000001
    - type: precision_at_3
      value: 9.611
    - type: precision_at_5
      value: 7.331
    - type: recall_at_1
      value: 17.279
    - type: recall_at_10
      value: 37.464999999999996
    - type: recall_at_100
      value: 56.458
    - type: recall_at_1000
      value: 76.759
    - type: recall_at_3
      value: 24.659
    - type: recall_at_5
      value: 30.672
    - type: map_at_1
      value: 14.901
    - type: map_at_10
      value: 20.268
    - type: map_at_100
      value: 21.143
    - type: map_at_1000
      value: 21.264
    - type: map_at_3
      value: 18.557000000000002
    - type: map_at_5
      value: 19.483
    - type: mrr_at_1
      value: 17.997
    - type: mrr_at_10
      value: 23.591
    - type: mrr_at_100
      value: 24.387
    - type: mrr_at_1000
      value: 24.471
    - type: mrr_at_3
      value: 21.874
    - type: mrr_at_5
      value: 22.797
    - type: ndcg_at_1
      value: 17.997
    - type: ndcg_at_10
      value: 23.87
    - type: ndcg_at_100
      value: 28.459
    - type: ndcg_at_1000
      value: 31.66
    - type: ndcg_at_3
      value: 20.779
    - type: ndcg_at_5
      value: 22.137
    - type: precision_at_1
      value: 17.997
    - type: precision_at_10
      value: 4.25
    - type: precision_at_100
      value: 0.761
    - type: precision_at_1000
      value: 0.121
    - type: precision_at_3
      value: 9.716
    - type: precision_at_5
      value: 6.909999999999999
    - type: recall_at_1
      value: 14.901
    - type: recall_at_10
      value: 31.44
    - type: recall_at_100
      value: 52.717000000000006
    - type: recall_at_1000
      value: 76.102
    - type: recall_at_3
      value: 22.675
    - type: recall_at_5
      value: 26.336
    - type: map_at_1
      value: 21.52
    - type: map_at_10
      value: 28.397
    - type: map_at_100
      value: 29.443
    - type: map_at_1000
      value: 29.56
    - type: map_at_3
      value: 26.501
    - type: map_at_5
      value: 27.375
    - type: mrr_at_1
      value: 25.28
    - type: mrr_at_10
      value: 32.102000000000004
    - type: mrr_at_100
      value: 33.005
    - type: mrr_at_1000
      value: 33.084
    - type: mrr_at_3
      value: 30.208000000000002
    - type: mrr_at_5
      value: 31.146
    - type: ndcg_at_1
      value: 25.28
    - type: ndcg_at_10
      value: 32.635
    - type: ndcg_at_100
      value: 37.672
    - type: ndcg_at_1000
      value: 40.602
    - type: ndcg_at_3
      value: 28.951999999999998
    - type: ndcg_at_5
      value: 30.336999999999996
    - type: precision_at_1
      value: 25.28
    - type: precision_at_10
      value: 5.3260000000000005
    - type: precision_at_100
      value: 0.8840000000000001
    - type: precision_at_1000
      value: 0.126
    - type: precision_at_3
      value: 12.687000000000001
    - type: precision_at_5
      value: 8.638
    - type: recall_at_1
      value: 21.52
    - type: recall_at_10
      value: 41.955
    - type: recall_at_100
      value: 64.21
    - type: recall_at_1000
      value: 85.28099999999999
    - type: recall_at_3
      value: 31.979999999999997
    - type: recall_at_5
      value: 35.406
    - type: map_at_1
      value: 20.296
    - type: map_at_10
      value: 28.449999999999996
    - type: map_at_100
      value: 29.847
    - type: map_at_1000
      value: 30.073
    - type: map_at_3
      value: 25.995
    - type: map_at_5
      value: 27.603
    - type: mrr_at_1
      value: 25.296000000000003
    - type: mrr_at_10
      value: 32.751999999999995
    - type: mrr_at_100
      value: 33.705
    - type: mrr_at_1000
      value: 33.783
    - type: mrr_at_3
      value: 30.731
    - type: mrr_at_5
      value: 32.006
    - type: ndcg_at_1
      value: 25.296000000000003
    - type: ndcg_at_10
      value: 33.555
    - type: ndcg_at_100
      value: 38.891999999999996
    - type: ndcg_at_1000
      value: 42.088
    - type: ndcg_at_3
      value: 29.944
    - type: ndcg_at_5
      value: 31.997999999999998
    - type: precision_at_1
      value: 25.296000000000003
    - type: precision_at_10
      value: 6.542000000000001
    - type: precision_at_100
      value: 1.354
    - type: precision_at_1000
      value: 0.22599999999999998
    - type: precision_at_3
      value: 14.360999999999999
    - type: precision_at_5
      value: 10.593
    - type: recall_at_1
      value: 20.296
    - type: recall_at_10
      value: 42.742000000000004
    - type: recall_at_100
      value: 67.351
    - type: recall_at_1000
      value: 88.774
    - type: recall_at_3
      value: 32.117000000000004
    - type: recall_at_5
      value: 37.788
    - type: map_at_1
      value: 18.157999999999998
    - type: map_at_10
      value: 24.342
    - type: map_at_100
      value: 25.201
    - type: map_at_1000
      value: 25.317
    - type: map_at_3
      value: 22.227
    - type: map_at_5
      value: 23.372999999999998
    - type: mrr_at_1
      value: 19.778000000000002
    - type: mrr_at_10
      value: 26.066
    - type: mrr_at_100
      value: 26.935
    - type: mrr_at_1000
      value: 27.022000000000002
    - type: mrr_at_3
      value: 24.214
    - type: mrr_at_5
      value: 25.268
    - type: ndcg_at_1
      value: 19.778000000000002
    - type: ndcg_at_10
      value: 28.104000000000003
    - type: ndcg_at_100
      value: 32.87
    - type: ndcg_at_1000
      value: 35.858000000000004
    - type: ndcg_at_3
      value: 24.107
    - type: ndcg_at_5
      value: 26.007
    - type: precision_at_1
      value: 19.778000000000002
    - type: precision_at_10
      value: 4.417999999999999
    - type: precision_at_100
      value: 0.739
    - type: precision_at_1000
      value: 0.109
    - type: precision_at_3
      value: 10.228
    - type: precision_at_5
      value: 7.172000000000001
    - type: recall_at_1
      value: 18.157999999999998
    - type: recall_at_10
      value: 37.967
    - type: recall_at_100
      value: 60.806000000000004
    - type: recall_at_1000
      value: 83.097
    - type: recall_at_3
      value: 27.223999999999997
    - type: recall_at_5
      value: 31.968000000000004
  - task:
      type: Retrieval
    dataset:
      name: MTEB ClimateFEVER
      type: climate-fever
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 7.055
    - type: map_at_10
      value: 11.609
    - type: map_at_100
      value: 12.83
    - type: map_at_1000
      value: 12.995000000000001
    - type: map_at_3
      value: 9.673
    - type: map_at_5
      value: 10.761999999999999
    - type: mrr_at_1
      value: 15.309000000000001
    - type: mrr_at_10
      value: 23.655
    - type: mrr_at_100
      value: 24.785
    - type: mrr_at_1000
      value: 24.856
    - type: mrr_at_3
      value: 20.499000000000002
    - type: mrr_at_5
      value: 22.425
    - type: ndcg_at_1
      value: 15.309000000000001
    - type: ndcg_at_10
      value: 17.252000000000002
    - type: ndcg_at_100
      value: 22.976
    - type: ndcg_at_1000
      value: 26.480999999999998
    - type: ndcg_at_3
      value: 13.418
    - type: ndcg_at_5
      value: 15.084
    - type: precision_at_1
      value: 15.309000000000001
    - type: precision_at_10
      value: 5.309
    - type: precision_at_100
      value: 1.1320000000000001
    - type: precision_at_1000
      value: 0.17600000000000002
    - type: precision_at_3
      value: 9.62
    - type: precision_at_5
      value: 7.883
    - type: recall_at_1
      value: 7.055
    - type: recall_at_10
      value: 21.891
    - type: recall_at_100
      value: 41.979
    - type: recall_at_1000
      value: 62.239999999999995
    - type: recall_at_3
      value: 12.722
    - type: recall_at_5
      value: 16.81
  - task:
      type: Retrieval
    dataset:
      name: MTEB DBPedia
      type: dbpedia-entity
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 6.909
    - type: map_at_10
      value: 12.844
    - type: map_at_100
      value: 16.435
    - type: map_at_1000
      value: 17.262
    - type: map_at_3
      value: 10.131
    - type: map_at_5
      value: 11.269
    - type: mrr_at_1
      value: 54.50000000000001
    - type: mrr_at_10
      value: 62.202
    - type: mrr_at_100
      value: 62.81
    - type: mrr_at_1000
      value: 62.824000000000005
    - type: mrr_at_3
      value: 60.5
    - type: mrr_at_5
      value: 61.324999999999996
    - type: ndcg_at_1
      value: 42.125
    - type: ndcg_at_10
      value: 28.284
    - type: ndcg_at_100
      value: 30.444
    - type: ndcg_at_1000
      value: 36.397
    - type: ndcg_at_3
      value: 33.439
    - type: ndcg_at_5
      value: 30.473
    - type: precision_at_1
      value: 54.50000000000001
    - type: precision_at_10
      value: 21.4
    - type: precision_at_100
      value: 6.192
    - type: precision_at_1000
      value: 1.398
    - type: precision_at_3
      value: 36.583
    - type: precision_at_5
      value: 28.799999999999997
    - type: recall_at_1
      value: 6.909
    - type: recall_at_10
      value: 17.296
    - type: recall_at_100
      value: 33.925
    - type: recall_at_1000
      value: 53.786
    - type: recall_at_3
      value: 11.333
    - type: recall_at_5
      value: 13.529
  - task:
      type: Classification
    dataset:
      name: MTEB EmotionClassification
      type: mteb/emotion
      config: default
      split: test
      revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
    metrics:
    - type: accuracy
      value: 36.08
    - type: f1
      value: 33.016420191943766
  - task:
      type: Retrieval
    dataset:
      name: MTEB FEVER
      type: fever
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 52.605000000000004
    - type: map_at_10
      value: 63.31400000000001
    - type: map_at_100
      value: 63.678000000000004
    - type: map_at_1000
      value: 63.699
    - type: map_at_3
      value: 61.141
    - type: map_at_5
      value: 62.517999999999994
    - type: mrr_at_1
      value: 56.871
    - type: mrr_at_10
      value: 67.915
    - type: mrr_at_100
      value: 68.24900000000001
    - type: mrr_at_1000
      value: 68.262
    - type: mrr_at_3
      value: 65.809
    - type: mrr_at_5
      value: 67.171
    - type: ndcg_at_1
      value: 56.871
    - type: ndcg_at_10
      value: 69.122
    - type: ndcg_at_100
      value: 70.855
    - type: ndcg_at_1000
      value: 71.368
    - type: ndcg_at_3
      value: 64.974
    - type: ndcg_at_5
      value: 67.318
    - type: precision_at_1
      value: 56.871
    - type: precision_at_10
      value: 9.029
    - type: precision_at_100
      value: 0.996
    - type: precision_at_1000
      value: 0.105
    - type: precision_at_3
      value: 25.893
    - type: precision_at_5
      value: 16.838
    - type: recall_at_1
      value: 52.605000000000004
    - type: recall_at_10
      value: 82.679
    - type: recall_at_100
      value: 90.586
    - type: recall_at_1000
      value: 94.38
    - type: recall_at_3
      value: 71.447
    - type: recall_at_5
      value: 77.218
  - task:
      type: Retrieval
    dataset:
      name: MTEB FiQA2018
      type: fiqa
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 10.759
    - type: map_at_10
      value: 18.877
    - type: map_at_100
      value: 20.498
    - type: map_at_1000
      value: 20.682000000000002
    - type: map_at_3
      value: 16.159000000000002
    - type: map_at_5
      value: 17.575
    - type: mrr_at_1
      value: 22.531000000000002
    - type: mrr_at_10
      value: 31.155
    - type: mrr_at_100
      value: 32.188
    - type: mrr_at_1000
      value: 32.245000000000005
    - type: mrr_at_3
      value: 28.781000000000002
    - type: mrr_at_5
      value: 30.054
    - type: ndcg_at_1
      value: 22.531000000000002
    - type: ndcg_at_10
      value: 25.189
    - type: ndcg_at_100
      value: 31.958
    - type: ndcg_at_1000
      value: 35.693999999999996
    - type: ndcg_at_3
      value: 22.235
    - type: ndcg_at_5
      value: 23.044999999999998
    - type: precision_at_1
      value: 22.531000000000002
    - type: precision_at_10
      value: 7.438000000000001
    - type: precision_at_100
      value: 1.418
    - type: precision_at_1000
      value: 0.208
    - type: precision_at_3
      value: 15.329
    - type: precision_at_5
      value: 11.451
    - type: recall_at_1
      value: 10.759
    - type: recall_at_10
      value: 31.416
    - type: recall_at_100
      value: 56.989000000000004
    - type: recall_at_1000
      value: 80.33200000000001
    - type: recall_at_3
      value: 20.61
    - type: recall_at_5
      value: 24.903
  - task:
      type: Retrieval
    dataset:
      name: MTEB HotpotQA
      type: hotpotqa
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 29.21
    - type: map_at_10
      value: 38.765
    - type: map_at_100
      value: 39.498
    - type: map_at_1000
      value: 39.568
    - type: map_at_3
      value: 36.699
    - type: map_at_5
      value: 37.925
    - type: mrr_at_1
      value: 58.42
    - type: mrr_at_10
      value: 65.137
    - type: mrr_at_100
      value: 65.542
    - type: mrr_at_1000
      value: 65.568
    - type: mrr_at_3
      value: 63.698
    - type: mrr_at_5
      value: 64.575
    - type: ndcg_at_1
      value: 58.42
    - type: ndcg_at_10
      value: 47.476
    - type: ndcg_at_100
      value: 50.466
    - type: ndcg_at_1000
      value: 52.064
    - type: ndcg_at_3
      value: 43.986
    - type: ndcg_at_5
      value: 45.824
    - type: precision_at_1
      value: 58.42
    - type: precision_at_10
      value: 9.649000000000001
    - type: precision_at_100
      value: 1.201
    - type: precision_at_1000
      value: 0.14100000000000001
    - type: precision_at_3
      value: 26.977
    - type: precision_at_5
      value: 17.642
    - type: recall_at_1
      value: 29.21
    - type: recall_at_10
      value: 48.244
    - type: recall_at_100
      value: 60.041
    - type: recall_at_1000
      value: 70.743
    - type: recall_at_3
      value: 40.466
    - type: recall_at_5
      value: 44.105
  - task:
      type: Classification
    dataset:
      name: MTEB ImdbClassification
      type: mteb/imdb
      config: default
      split: test
      revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
    metrics:
    - type: accuracy
      value: 58.7064
    - type: ap
      value: 55.36326227125519
    - type: f1
      value: 57.46763115215848
  - task:
      type: Retrieval
    dataset:
      name: MTEB MSMARCO
      type: msmarco
      config: default
      split: dev
      revision: None
    metrics:
    - type: map_at_1
      value: 15.889000000000001
    - type: map_at_10
      value: 25.979000000000003
    - type: map_at_100
      value: 27.21
    - type: map_at_1000
      value: 27.284000000000002
    - type: map_at_3
      value: 22.665
    - type: map_at_5
      value: 24.578
    - type: mrr_at_1
      value: 16.39
    - type: mrr_at_10
      value: 26.504
    - type: mrr_at_100
      value: 27.689999999999998
    - type: mrr_at_1000
      value: 27.758
    - type: mrr_at_3
      value: 23.24
    - type: mrr_at_5
      value: 25.108000000000004
    - type: ndcg_at_1
      value: 16.39
    - type: ndcg_at_10
      value: 31.799
    - type: ndcg_at_100
      value: 38.034
    - type: ndcg_at_1000
      value: 39.979
    - type: ndcg_at_3
      value: 25.054
    - type: ndcg_at_5
      value: 28.463
    - type: precision_at_1
      value: 16.39
    - type: precision_at_10
      value: 5.189
    - type: precision_at_100
      value: 0.835
    - type: precision_at_1000
      value: 0.1
    - type: precision_at_3
      value: 10.84
    - type: precision_at_5
      value: 8.238
    - type: recall_at_1
      value: 15.889000000000001
    - type: recall_at_10
      value: 49.739
    - type: recall_at_100
      value: 79.251
    - type: recall_at_1000
      value: 94.298
    - type: recall_at_3
      value: 31.427
    - type: recall_at_5
      value: 39.623000000000005
  - task:
      type: Classification
    dataset:
      name: MTEB MTOPDomainClassification (en)
      type: mteb/mtop_domain
      config: en
      split: test
      revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
    metrics:
    - type: accuracy
      value: 88.81668946648426
    - type: f1
      value: 88.55200075528438
  - task:
      type: Classification
    dataset:
      name: MTEB MTOPIntentClassification (en)
      type: mteb/mtop_intent
      config: en
      split: test
      revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
    metrics:
    - type: accuracy
      value: 58.611491108071135
    - type: f1
      value: 42.12391403999353
  - task:
      type: Classification
    dataset:
      name: MTEB MassiveIntentClassification (en)
      type: mteb/amazon_massive_intent
      config: en
      split: test
      revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
    metrics:
    - type: accuracy
      value: 64.67047747141896
    - type: f1
      value: 62.88410885922258
  - task:
      type: Classification
    dataset:
      name: MTEB MassiveScenarioClassification (en)
      type: mteb/amazon_massive_scenario
      config: en
      split: test
      revision: 7d571f92784cd94a019292a1f45445077d0ef634
    metrics:
    - type: accuracy
      value: 71.78547410894419
    - type: f1
      value: 71.69467869218154
  - task:
      type: Clustering
    dataset:
      name: MTEB MedrxivClusteringP2P
      type: mteb/medrxiv-clustering-p2p
      config: default
      split: test
      revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
    metrics:
    - type: v_measure
      value: 27.23799937752035
  - task:
      type: Clustering
    dataset:
      name: MTEB MedrxivClusteringS2S
      type: mteb/medrxiv-clustering-s2s
      config: default
      split: test
      revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
    metrics:
    - type: v_measure
      value: 23.26502601343789
  - task:
      type: Reranking
    dataset:
      name: MTEB MindSmallReranking
      type: mteb/mind_small
      config: default
      split: test
      revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
    metrics:
    - type: map
      value: 30.680711484149832
    - type: mrr
      value: 31.705059795117307
  - task:
      type: Retrieval
    dataset:
      name: MTEB NFCorpus
      type: nfcorpus
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 4.077
    - type: map_at_10
      value: 8.657
    - type: map_at_100
      value: 10.753
    - type: map_at_1000
      value: 11.885
    - type: map_at_3
      value: 6.5089999999999995
    - type: map_at_5
      value: 7.405
    - type: mrr_at_1
      value: 38.7
    - type: mrr_at_10
      value: 46.065
    - type: mrr_at_100
      value: 46.772000000000006
    - type: mrr_at_1000
      value: 46.83
    - type: mrr_at_3
      value: 44.118
    - type: mrr_at_5
      value: 45.015
    - type: ndcg_at_1
      value: 36.997
    - type: ndcg_at_10
      value: 25.96
    - type: ndcg_at_100
      value: 23.607
    - type: ndcg_at_1000
      value: 32.317
    - type: ndcg_at_3
      value: 31.06
    - type: ndcg_at_5
      value: 28.921000000000003
    - type: precision_at_1
      value: 38.7
    - type: precision_at_10
      value: 19.195
    - type: precision_at_100
      value: 6.164
    - type: precision_at_1000
      value: 1.839
    - type: precision_at_3
      value: 28.999000000000002
    - type: precision_at_5
      value: 25.014999999999997
    - type: recall_at_1
      value: 4.077
    - type: recall_at_10
      value: 11.802
    - type: recall_at_100
      value: 24.365000000000002
    - type: recall_at_1000
      value: 55.277
    - type: recall_at_3
      value: 7.435
    - type: recall_at_5
      value: 8.713999999999999
  - task:
      type: Retrieval
    dataset:
      name: MTEB NQ
      type: nq
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 19.588
    - type: map_at_10
      value: 32.08
    - type: map_at_100
      value: 33.32
    - type: map_at_1000
      value: 33.377
    - type: map_at_3
      value: 28.166000000000004
    - type: map_at_5
      value: 30.383
    - type: mrr_at_1
      value: 22.161
    - type: mrr_at_10
      value: 34.121
    - type: mrr_at_100
      value: 35.171
    - type: mrr_at_1000
      value: 35.214
    - type: mrr_at_3
      value: 30.692000000000004
    - type: mrr_at_5
      value: 32.706
    - type: ndcg_at_1
      value: 22.131999999999998
    - type: ndcg_at_10
      value: 38.887
    - type: ndcg_at_100
      value: 44.433
    - type: ndcg_at_1000
      value: 45.823
    - type: ndcg_at_3
      value: 31.35
    - type: ndcg_at_5
      value: 35.144
    - type: precision_at_1
      value: 22.131999999999998
    - type: precision_at_10
      value: 6.8629999999999995
    - type: precision_at_100
      value: 0.993
    - type: precision_at_1000
      value: 0.11199999999999999
    - type: precision_at_3
      value: 14.706
    - type: precision_at_5
      value: 10.972999999999999
    - type: recall_at_1
      value: 19.588
    - type: recall_at_10
      value: 57.703
    - type: recall_at_100
      value: 82.194
    - type: recall_at_1000
      value: 92.623
    - type: recall_at_3
      value: 38.012
    - type: recall_at_5
      value: 46.847
  - task:
      type: Retrieval
    dataset:
      name: MTEB QuoraRetrieval
      type: quora
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 68.038
    - type: map_at_10
      value: 81.572
    - type: map_at_100
      value: 82.25200000000001
    - type: map_at_1000
      value: 82.27600000000001
    - type: map_at_3
      value: 78.618
    - type: map_at_5
      value: 80.449
    - type: mrr_at_1
      value: 78.31
    - type: mrr_at_10
      value: 84.98
    - type: mrr_at_100
      value: 85.122
    - type: mrr_at_1000
      value: 85.124
    - type: mrr_at_3
      value: 83.852
    - type: mrr_at_5
      value: 84.6
    - type: ndcg_at_1
      value: 78.31
    - type: ndcg_at_10
      value: 85.693
    - type: ndcg_at_100
      value: 87.191
    - type: ndcg_at_1000
      value: 87.386
    - type: ndcg_at_3
      value: 82.585
    - type: ndcg_at_5
      value: 84.255
    - type: precision_at_1
      value: 78.31
    - type: precision_at_10
      value: 12.986
    - type: precision_at_100
      value: 1.505
    - type: precision_at_1000
      value: 0.156
    - type: precision_at_3
      value: 36.007
    - type: precision_at_5
      value: 23.735999999999997
    - type: recall_at_1
      value: 68.038
    - type: recall_at_10
      value: 93.598
    - type: recall_at_100
      value: 98.869
    - type: recall_at_1000
      value: 99.86500000000001
    - type: recall_at_3
      value: 84.628
    - type: recall_at_5
      value: 89.316
  - task:
      type: Clustering
    dataset:
      name: MTEB RedditClustering
      type: mteb/reddit-clustering
      config: default
      split: test
      revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
    metrics:
    - type: v_measure
      value: 37.948231664922865
  - task:
      type: Clustering
    dataset:
      name: MTEB RedditClusteringP2P
      type: mteb/reddit-clustering-p2p
      config: default
      split: test
      revision: 282350215ef01743dc01b456c7f5241fa8937f16
    metrics:
    - type: v_measure
      value: 49.90597913763894
  - task:
      type: Retrieval
    dataset:
      name: MTEB SCIDOCS
      type: scidocs
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 3.753
    - type: map_at_10
      value: 8.915
    - type: map_at_100
      value: 10.374
    - type: map_at_1000
      value: 10.612
    - type: map_at_3
      value: 6.577
    - type: map_at_5
      value: 7.8
    - type: mrr_at_1
      value: 18.4
    - type: mrr_at_10
      value: 27.325
    - type: mrr_at_100
      value: 28.419
    - type: mrr_at_1000
      value: 28.494000000000003
    - type: mrr_at_3
      value: 24.349999999999998
    - type: mrr_at_5
      value: 26.205000000000002
    - type: ndcg_at_1
      value: 18.4
    - type: ndcg_at_10
      value: 15.293000000000001
    - type: ndcg_at_100
      value: 21.592
    - type: ndcg_at_1000
      value: 26.473000000000003
    - type: ndcg_at_3
      value: 14.748
    - type: ndcg_at_5
      value: 12.98
    - type: precision_at_1
      value: 18.4
    - type: precision_at_10
      value: 7.779999999999999
    - type: precision_at_100
      value: 1.693
    - type: precision_at_1000
      value: 0.28800000000000003
    - type: precision_at_3
      value: 13.700000000000001
    - type: precision_at_5
      value: 11.379999999999999
    - type: recall_at_1
      value: 3.753
    - type: recall_at_10
      value: 15.806999999999999
    - type: recall_at_100
      value: 34.37
    - type: recall_at_1000
      value: 58.463
    - type: recall_at_3
      value: 8.338
    - type: recall_at_5
      value: 11.538
  - task:
      type: STS
    dataset:
      name: MTEB SICK-R
      type: mteb/sickr-sts
      config: default
      split: test
      revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
    metrics:
    - type: cos_sim_pearson
      value: 82.58843987639705
    - type: cos_sim_spearman
      value: 76.33071660715956
    - type: euclidean_pearson
      value: 72.8029921002978
    - type: euclidean_spearman
      value: 69.34534284782808
    - type: manhattan_pearson
      value: 72.49781034973653
    - type: manhattan_spearman
      value: 69.24754112621694
  - task:
      type: STS
    dataset:
      name: MTEB STS12
      type: mteb/sts12-sts
      config: default
      split: test
      revision: a0d554a64d88156834ff5ae9920b964011b16384
    metrics:
    - type: cos_sim_pearson
      value: 83.31673079903189
    - type: cos_sim_spearman
      value: 74.27699263517789
    - type: euclidean_pearson
      value: 69.4008910999579
    - type: euclidean_spearman
      value: 59.0716984643048
    - type: manhattan_pearson
      value: 68.87342686919199
    - type: manhattan_spearman
      value: 58.904612865335025
  - task:
      type: STS
    dataset:
      name: MTEB STS13
      type: mteb/sts13-sts
      config: default
      split: test
      revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
    metrics:
    - type: cos_sim_pearson
      value: 77.59122302327788
    - type: cos_sim_spearman
      value: 78.55383586979005
    - type: euclidean_pearson
      value: 68.18338642204289
    - type: euclidean_spearman
      value: 68.95092864180276
    - type: manhattan_pearson
      value: 68.08807059822706
    - type: manhattan_spearman
      value: 68.86135938270193
  - task:
      type: STS
    dataset:
      name: MTEB STS14
      type: mteb/sts14-sts
      config: default
      split: test
      revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
    metrics:
    - type: cos_sim_pearson
      value: 78.51766841424501
    - type: cos_sim_spearman
      value: 73.84318001499558
    - type: euclidean_pearson
      value: 67.2007138855177
    - type: euclidean_spearman
      value: 63.98672842723766
    - type: manhattan_pearson
      value: 67.17773810895949
    - type: manhattan_spearman
      value: 64.07359154832962
  - task:
      type: STS
    dataset:
      name: MTEB STS15
      type: mteb/sts15-sts
      config: default
      split: test
      revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
    metrics:
    - type: cos_sim_pearson
      value: 82.73438541570299
    - type: cos_sim_spearman
      value: 83.71357922283677
    - type: euclidean_pearson
      value: 57.50131347498546
    - type: euclidean_spearman
      value: 57.73623619252132
    - type: manhattan_pearson
      value: 58.082992079000725
    - type: manhattan_spearman
      value: 58.42728201167522
  - task:
      type: STS
    dataset:
      name: MTEB STS16
      type: mteb/sts16-sts
      config: default
      split: test
      revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
    metrics:
    - type: cos_sim_pearson
      value: 78.14794654172421
    - type: cos_sim_spearman
      value: 80.025736165043
    - type: euclidean_pearson
      value: 65.87773913985473
    - type: euclidean_spearman
      value: 66.69337751784794
    - type: manhattan_pearson
      value: 66.01039761004415
    - type: manhattan_spearman
      value: 66.89215027952318
  - task:
      type: STS
    dataset:
      name: MTEB STS17 (en-en)
      type: mteb/sts17-crosslingual-sts
      config: en-en
      split: test
      revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
    metrics:
    - type: cos_sim_pearson
      value: 87.10554507136152
    - type: cos_sim_spearman
      value: 87.4898082140765
    - type: euclidean_pearson
      value: 72.19391114541367
    - type: euclidean_spearman
      value: 70.36647944993783
    - type: manhattan_pearson
      value: 72.18680758133698
    - type: manhattan_spearman
      value: 70.3871215447305
  - task:
      type: STS
    dataset:
      name: MTEB STS22 (en)
      type: mteb/sts22-crosslingual-sts
      config: en
      split: test
      revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
    metrics:
    - type: cos_sim_pearson
      value: 64.54868111501618
    - type: cos_sim_spearman
      value: 64.25173617448473
    - type: euclidean_pearson
      value: 39.116088900637116
    - type: euclidean_spearman
      value: 53.300772929884
    - type: manhattan_pearson
      value: 38.3844195287959
    - type: manhattan_spearman
      value: 52.846675312001246
  - task:
      type: STS
    dataset:
      name: MTEB STSBenchmark
      type: mteb/stsbenchmark-sts
      config: default
      split: test
      revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
    metrics:
    - type: cos_sim_pearson
      value: 80.04396610550214
    - type: cos_sim_spearman
      value: 79.19504854997832
    - type: euclidean_pearson
      value: 66.3284657637072
    - type: euclidean_spearman
      value: 63.69531796729492
    - type: manhattan_pearson
      value: 66.82324081038026
    - type: manhattan_spearman
      value: 64.18254512904923
  - task:
      type: Reranking
    dataset:
      name: MTEB SciDocsRR
      type: mteb/scidocs-reranking
      config: default
      split: test
      revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
    metrics:
    - type: map
      value: 74.16264051781705
    - type: mrr
      value: 91.80864796060874
  - task:
      type: Retrieval
    dataset:
      name: MTEB SciFact
      type: scifact
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 38.983000000000004
    - type: map_at_10
      value: 47.858000000000004
    - type: map_at_100
      value: 48.695
    - type: map_at_1000
      value: 48.752
    - type: map_at_3
      value: 45.444
    - type: map_at_5
      value: 46.906
    - type: mrr_at_1
      value: 41.333
    - type: mrr_at_10
      value: 49.935
    - type: mrr_at_100
      value: 50.51
    - type: mrr_at_1000
      value: 50.55500000000001
    - type: mrr_at_3
      value: 47.833
    - type: mrr_at_5
      value: 49.117
    - type: ndcg_at_1
      value: 41.333
    - type: ndcg_at_10
      value: 52.398999999999994
    - type: ndcg_at_100
      value: 56.196
    - type: ndcg_at_1000
      value: 57.838
    - type: ndcg_at_3
      value: 47.987
    - type: ndcg_at_5
      value: 50.356
    - type: precision_at_1
      value: 41.333
    - type: precision_at_10
      value: 7.167
    - type: precision_at_100
      value: 0.9299999999999999
    - type: precision_at_1000
      value: 0.108
    - type: precision_at_3
      value: 19.0
    - type: precision_at_5
      value: 12.8
    - type: recall_at_1
      value: 38.983000000000004
    - type: recall_at_10
      value: 64.183
    - type: recall_at_100
      value: 82.02199999999999
    - type: recall_at_1000
      value: 95.167
    - type: recall_at_3
      value: 52.383
    - type: recall_at_5
      value: 58.411
  - task:
      type: PairClassification
    dataset:
      name: MTEB SprintDuplicateQuestions
      type: mteb/sprintduplicatequestions-pairclassification
      config: default
      split: test
      revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
    metrics:
    - type: cos_sim_accuracy
      value: 99.8019801980198
    - type: cos_sim_ap
      value: 94.9287554635848
    - type: cos_sim_f1
      value: 89.83739837398375
    - type: cos_sim_precision
      value: 91.32231404958677
    - type: cos_sim_recall
      value: 88.4
    - type: dot_accuracy
      value: 99.23762376237623
    - type: dot_ap
      value: 55.22534191245801
    - type: dot_f1
      value: 54.054054054054056
    - type: dot_precision
      value: 55.15088449531738
    - type: dot_recall
      value: 53.0
    - type: euclidean_accuracy
      value: 99.6108910891089
    - type: euclidean_ap
      value: 82.5195111329438
    - type: euclidean_f1
      value: 78.2847718526663
    - type: euclidean_precision
      value: 86.93528693528694
    - type: euclidean_recall
      value: 71.2
    - type: manhattan_accuracy
      value: 99.5970297029703
    - type: manhattan_ap
      value: 81.96876777875492
    - type: manhattan_f1
      value: 77.33773377337734
    - type: manhattan_precision
      value: 85.94132029339853
    - type: manhattan_recall
      value: 70.3
    - type: max_accuracy
      value: 99.8019801980198
    - type: max_ap
      value: 94.9287554635848
    - type: max_f1
      value: 89.83739837398375
  - task:
      type: Clustering
    dataset:
      name: MTEB StackExchangeClustering
      type: mteb/stackexchange-clustering
      config: default
      split: test
      revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
    metrics:
    - type: v_measure
      value: 46.34997003954114
  - task:
      type: Clustering
    dataset:
      name: MTEB StackExchangeClusteringP2P
      type: mteb/stackexchange-clustering-p2p
      config: default
      split: test
      revision: 815ca46b2622cec33ccafc3735d572c266efdb44
    metrics:
    - type: v_measure
      value: 31.462336020554893
  - task:
      type: Reranking
    dataset:
      name: MTEB StackOverflowDupQuestions
      type: mteb/stackoverflowdupquestions-reranking
      config: default
      split: test
      revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
    metrics:
    - type: map
      value: 47.1757817459526
    - type: mrr
      value: 47.941057104660054
  - task:
      type: Summarization
    dataset:
      name: MTEB SummEval
      type: mteb/summeval
      config: default
      split: test
      revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
    metrics:
    - type: cos_sim_pearson
      value: 30.56106249068471
    - type: cos_sim_spearman
      value: 31.24613190558528
    - type: dot_pearson
      value: 20.486610035794257
    - type: dot_spearman
      value: 23.115667545894546
  - task:
      type: Retrieval
    dataset:
      name: MTEB TRECCOVID
      type: trec-covid
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 0.182
    - type: map_at_10
      value: 1.155
    - type: map_at_100
      value: 5.118
    - type: map_at_1000
      value: 11.827
    - type: map_at_3
      value: 0.482
    - type: map_at_5
      value: 0.712
    - type: mrr_at_1
      value: 70.0
    - type: mrr_at_10
      value: 79.483
    - type: mrr_at_100
      value: 79.637
    - type: mrr_at_1000
      value: 79.637
    - type: mrr_at_3
      value: 77.667
    - type: mrr_at_5
      value: 78.567
    - type: ndcg_at_1
      value: 63.0
    - type: ndcg_at_10
      value: 52.303
    - type: ndcg_at_100
      value: 37.361
    - type: ndcg_at_1000
      value: 32.84
    - type: ndcg_at_3
      value: 58.274
    - type: ndcg_at_5
      value: 55.601
    - type: precision_at_1
      value: 70.0
    - type: precision_at_10
      value: 55.60000000000001
    - type: precision_at_100
      value: 37.96
    - type: precision_at_1000
      value: 14.738000000000001
    - type: precision_at_3
      value: 62.666999999999994
    - type: precision_at_5
      value: 60.0
    - type: recall_at_1
      value: 0.182
    - type: recall_at_10
      value: 1.4120000000000001
    - type: recall_at_100
      value: 8.533
    - type: recall_at_1000
      value: 30.572
    - type: recall_at_3
      value: 0.5309999999999999
    - type: recall_at_5
      value: 0.814
  - task:
      type: Retrieval
    dataset:
      name: MTEB Touche2020
      type: webis-touche2020
      config: default
      split: test
      revision: None
    metrics:
    - type: map_at_1
      value: 1.385
    - type: map_at_10
      value: 7.185999999999999
    - type: map_at_100
      value: 11.642
    - type: map_at_1000
      value: 12.953000000000001
    - type: map_at_3
      value: 3.496
    - type: map_at_5
      value: 4.82
    - type: mrr_at_1
      value: 16.326999999999998
    - type: mrr_at_10
      value: 29.461
    - type: mrr_at_100
      value: 31.436999999999998
    - type: mrr_at_1000
      value: 31.436999999999998
    - type: mrr_at_3
      value: 24.490000000000002
    - type: mrr_at_5
      value: 27.857
    - type: ndcg_at_1
      value: 14.285999999999998
    - type: ndcg_at_10
      value: 16.672
    - type: ndcg_at_100
      value: 28.691
    - type: ndcg_at_1000
      value: 39.817
    - type: ndcg_at_3
      value: 15.277
    - type: ndcg_at_5
      value: 15.823
    - type: precision_at_1
      value: 16.326999999999998
    - type: precision_at_10
      value: 15.509999999999998
    - type: precision_at_100
      value: 6.49
    - type: precision_at_1000
      value: 1.4080000000000001
    - type: precision_at_3
      value: 16.326999999999998
    - type: precision_at_5
      value: 16.735
    - type: recall_at_1
      value: 1.385
    - type: recall_at_10
      value: 12.586
    - type: recall_at_100
      value: 40.765
    - type: recall_at_1000
      value: 75.198
    - type: recall_at_3
      value: 4.326
    - type: recall_at_5
      value: 7.074999999999999
  - task:
      type: Classification
    dataset:
      name: MTEB ToxicConversationsClassification
      type: mteb/toxic_conversations_50k
      config: default
      split: test
      revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
    metrics:
    - type: accuracy
      value: 59.4402
    - type: ap
      value: 10.16922814263879
    - type: f1
      value: 45.374485104940476
  - task:
      type: Classification
    dataset:
      name: MTEB TweetSentimentExtractionClassification
      type: mteb/tweet_sentiment_extraction
      config: default
      split: test
      revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
    metrics:
    - type: accuracy
      value: 54.25863044708545
    - type: f1
      value: 54.20154252609619
  - task:
      type: Clustering
    dataset:
      name: MTEB TwentyNewsgroupsClustering
      type: mteb/twentynewsgroups-clustering
      config: default
      split: test
      revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
    metrics:
    - type: v_measure
      value: 34.3883169293051
  - task:
      type: PairClassification
    dataset:
      name: MTEB TwitterSemEval2015
      type: mteb/twittersemeval2015-pairclassification
      config: default
      split: test
      revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
    metrics:
    - type: cos_sim_accuracy
      value: 81.76670441676104
    - type: cos_sim_ap
      value: 59.29878710961347
    - type: cos_sim_f1
      value: 57.33284971587474
    - type: cos_sim_precision
      value: 52.9122963624191
    - type: cos_sim_recall
      value: 62.559366754617415
    - type: dot_accuracy
      value: 77.52279907015557
    - type: dot_ap
      value: 34.17588904643467
    - type: dot_f1
      value: 41.063567529494634
    - type: dot_precision
      value: 30.813953488372093
    - type: dot_recall
      value: 61.53034300791557
    - type: euclidean_accuracy
      value: 80.61631996185254
    - type: euclidean_ap
      value: 54.00362361479352
    - type: euclidean_f1
      value: 53.99111751290361
    - type: euclidean_precision
      value: 49.52653600528518
    - type: euclidean_recall
      value: 59.340369393139845
    - type: manhattan_accuracy
      value: 80.65208320915539
    - type: manhattan_ap
      value: 54.18329507159467
    - type: manhattan_f1
      value: 53.85550960836779
    - type: manhattan_precision
      value: 49.954873646209386
    - type: manhattan_recall
      value: 58.41688654353562
    - type: max_accuracy
      value: 81.76670441676104
    - type: max_ap
      value: 59.29878710961347
    - type: max_f1
      value: 57.33284971587474
  - task:
      type: PairClassification
    dataset:
      name: MTEB TwitterURLCorpus
      type: mteb/twitterurlcorpus-pairclassification
      config: default
      split: test
      revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
    metrics:
    - type: cos_sim_accuracy
      value: 87.99433383785463
    - type: cos_sim_ap
      value: 83.43513915159009
    - type: cos_sim_f1
      value: 76.3906784964842
    - type: cos_sim_precision
      value: 73.19223985890653
    - type: cos_sim_recall
      value: 79.88142901139513
    - type: dot_accuracy
      value: 81.96142352621571
    - type: dot_ap
      value: 67.78764755689359
    - type: dot_f1
      value: 64.42823356983445
    - type: dot_precision
      value: 56.77801913931779
    - type: dot_recall
      value: 74.46104096088698
    - type: euclidean_accuracy
      value: 81.9478402607987
    - type: euclidean_ap
      value: 67.13958457373279
    - type: euclidean_f1
      value: 60.45118343195266
    - type: euclidean_precision
      value: 58.1625391403359
    - type: euclidean_recall
      value: 62.92731752386819
    - type: manhattan_accuracy
      value: 82.01769705437188
    - type: manhattan_ap
      value: 67.24709477497046
    - type: manhattan_f1
      value: 60.4103846436714
    - type: manhattan_precision
      value: 57.82063916654935
    - type: manhattan_recall
      value: 63.24299353249153
    - type: max_accuracy
      value: 87.99433383785463
    - type: max_ap
      value: 83.43513915159009
    - type: max_f1
      value: 76.3906784964842
---
# # Fast-Inference with Ctranslate2
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.

quantized version of [jinaai/jina-embedding-s-en-v1](https://huggingface.co/jinaai/jina-embedding-s-en-v1)
```bash
pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.17.1
```

```python
# from transformers import AutoTokenizer
model_name = "michaelfeil/ct2fast-jina-embedding-s-en-v1"
model_name_orig="jinaai/jina-embedding-s-en-v1"

from hf_hub_ctranslate2 import EncoderCT2fromHfHub
model = EncoderCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name,
        device="cuda",
        compute_type="int8_float16"
)
outputs = model.generate(
    text=["I like soccer", "I like tennis", "The eiffel tower is in Paris"],
    max_length=64,
) # perform downstream tasks on outputs
outputs["pooler_output"]
outputs["last_hidden_state"]
outputs["attention_mask"]

# alternative, use SentenceTransformer Mix-In
# for end-to-end Sentence embeddings generation
# (not pulling from this CT2fast-HF repo)

from hf_hub_ctranslate2 import CT2SentenceTransformer
model = CT2SentenceTransformer(
    model_name_orig, compute_type="int8_float16", device="cuda"
)
embeddings = model.encode(
    ["I like soccer", "I like tennis", "The eiffel tower is in Paris"],
    batch_size=32,
    convert_to_numpy=True,
    normalize_embeddings=True,
)
print(embeddings.shape, embeddings)
scores = (embeddings @ embeddings.T) * 100

# Hint: you can also host this code via REST API and
# via github.com/michaelfeil/infinity  


```

Checkpoint compatible to [ctranslate2>=3.17.1](https://github.com/OpenNMT/CTranslate2)
and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2)
- `compute_type=int8_float16` for `device="cuda"`
- `compute_type=int8`  for `device="cpu"`

Converted on 2023-10-13 using
```
LLama-2 -> removed <pad> token.
```

# Licence and other remarks:
This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

# Original description
    

<br><br>

<p align="center">
<img src="https://github.com/jina-ai/finetuner/blob/main/docs/_static/finetuner-logo-ani.svg?raw=true" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px">
</p>


<p align="center">
<b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>, <a href="https://github.com/jina-ai/finetuner"><b>Finetuner</b></a> team.</b>
</p>


## Intented Usage & Model Info

`jina-embedding-s-en-v1` is a language model that has been trained using Jina AI's Linnaeus-Clean dataset.
This dataset consists of 380 million pairs of sentences, which include both query-document pairs.
These pairs were obtained from various domains and were carefully selected through a thorough cleaning process.
The Linnaeus-Full dataset, from which the Linnaeus-Clean dataset is derived, originally contained 1.6 billion sentence pairs.

The model has a range of use cases, including information retrieval, semantic textual similarity, text reranking, and more.

With a compact size of just 35 million parameters,
the model enables lightning-fast inference while still delivering impressive performance.
Additionally, we provide the following options:

- [`jina-embedding-t-en-v1`](https://huggingface.co/jinaai/jina-embedding-t-en-v1): 14 million parameters.
- [`jina-embedding-s-en-v1`](https://huggingface.co/jinaai/jina-embedding-s-en-v1): 35 million parameters **(you are here)**.
- [`jina-embedding-b-en-v1`](https://huggingface.co/jinaai/jina-embedding-b-en-v1): 110 million parameters.
- [`jina-embedding-l-en-v1`](https://huggingface.co/jinaai/jina-embedding-l-en-v1): 330 million parameters.
- `jina-embedding-1b-en-v1`: 1.2 billion parameters, 10 times bert-base (soon).
- `jina-embedding-6b-en-v1`: 6 billion parameters, 30 times bert-base (soon).

## Data & Parameters

Please checkout our [technical blog](https://arxiv.org/abs/2307.11224).

## Metrics

We compared the model against `all-minilm-l6-v2`/`all-mpnet-base-v2` from sbert and `text-embeddings-ada-002` from OpenAI:

|Name|param    |dimension|
|------------------------------|-----|------|
|all-minilm-l6-v2|23m      |384|
|all-mpnet-base-v2 |110m     |768|
|ada-embedding-002|Unknown/OpenAI API  |1536|
|jina-embedding-t-en-v1|14m      |312|
|jina-embedding-s-en-v1|35m      |512|
|jina-embedding-b-en-v1|110m      |768|
|jina-embedding-l-en-v1|330m      |1024|


|Name|STS12|STS13|STS14|STS15|STS16|STS17|TRECOVID|Quora|SciFact|
|------------------------------|-----|-----|-----|-----|-----|-----|--------|-----|-----|
|all-minilm-l6-v2|0.724|0.806|0.756|0.854|0.79 |0.876|0.473   |0.876|0.645  |
|all-mpnet-base-v2|0.726|**0.835**|0.78 |0.857|0.8  |**0.906**|0.513   |0.875|0.656  |
|ada-embedding-002|0.698|0.833|0.761|0.861|**0.86** |0.903|**0.685**   |0.876|**0.726**  |
|jina-embedding-t-en-v1|0.717|0.773|0.731|0.829|0.777|0.860|0.482   |0.840|0.522  |
|jina-embedding-s-en-v1|0.743|0.786|0.738|0.837|0.80|0.875|0.523   |0.857|0.524  |
|jina-embedding-b-en-v1|**0.751**|0.809|0.761|0.856|0.812|0.890|0.606   |0.876|0.594  |
|jina-embedding-l-en-v1|0.745|0.832|**0.781**|**0.869**|0.837|0.902|0.573   |**0.881**|0.598  |

## Usage

Use with Jina AI Finetuner

```python
!pip install finetuner
import finetuner

model = finetuner.build_model('jinaai/jina-embedding-s-en-v1')
embeddings = finetuner.encode(
    model=model,
    data=['how is the weather today', 'What is the current weather like today?']
)
print(finetuner.cos_sim(embeddings[0], embeddings[1]))
```

Use with sentence-transformers:

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

sentences = ['how is the weather today', 'What is the current weather like today?']

model = SentenceTransformer('jinaai/jina-embedding-s-en-v1')
embeddings = model.encode(sentences)
print(cos_sim(embeddings[0], embeddings[1]))
```

## Fine-tuning

Please consider [Finetuner](https://github.com/jina-ai/finetuner).

## Plans

1. The development of `jina-embedding-s-en-v2` is currently underway with two main objectives: improving performance and increasing the maximum sequence length.
2. We are currently working on a bilingual embedding model that combines English and X language. The upcoming model will be called `jina-embedding-s/b/l-de-v1`.

## Contact

Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas.

## Citation

If you find Jina Embeddings useful in your research, please cite the following paper:

``` latex
@misc{günther2023jina,
      title={Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models}, 
      author={Michael Günther and Louis Milliken and Jonathan Geuter and Georgios Mastrapas and Bo Wang and Han Xiao},
      year={2023},
      eprint={2307.11224},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```