chelleboyer/llm-evals-2-a56b96e9-5b1a-4351-9b07-3c46a9e2bfe6

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("chelleboyer/llm-evals-2-a56b96e9-5b1a-4351-9b07-3c46a9e2bfe6")
# Run inference
sentences = [
    'What is the expression for the minimum variance of $z^{mathsfcv;alpha}z^\{\backslash \backslash mathsf\{cv\};\backslash \backslash alpha\}$ in terms of $rho\backslash \backslash rho$ and $mathrmVar[z]\backslash \backslash mathrm\{Var\}[z]$?',
    'minα∈ℝ\u2061Var\u2062[z𝖼𝗏;α]=(1−ρ2)\u2062Var\u2062[z].subscript𝛼ℝVardelimited-[]superscript𝑧𝖼𝗏𝛼1superscript𝜌2Vardelimited-[]𝑧\\displaystyle\\min_{\\alpha\\in\\mathbb{R}}\\mathrm{Var}[z^{\\mathsf{cv};\\alpha}]=%\n\\left(1-\\rho^{2}\\right)\\mathrm{Var}[z].roman_min start_POSTSUBSCRIPT italic_α ∈ blackboard_R end_POSTSUBSCRIPT roman_Var [ italic_z start_POSTSUPERSCRIPT sansserif_cv ; italic_α end_POSTSUPERSCRIPT ] = ( 1 - italic_ρ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) roman_Var [ italic_z ] .\n\n\n\nThe minimum is achieved if and only if α𝛼\\alphaitalic_α equals',
    'explored how to select these components or how their different combinations influence the results.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.94
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.94
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.94
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9752
cosine_mrr@10	0.9667
cosine_map@100	0.9667

Training Details

Training Dataset

Unnamed Dataset

• Size: 782 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 782 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 5 tokens mean: 31.75 tokens max: 178 tokens	min: 3 tokens mean: 148.03 tokens max: 309 tokens

• Samples:

  sentence_0	sentence_1
  What role do control variates play in accelerating unbiased LLM evaluation as discussed in the context?	Accelerating Unbiased LLM Evaluation via Synthetic Feedback 1 Introduction 2 Related Work 2.1 LLM Evaluation: Metric, Benchmark and Systems 2.2 Speeding Up LLM Evaluation 2.3 Control Variates, Application, and related techniques 3 Preliminaries 3.1 LLM Evaluation 3.2 Human and Synthetic Evaluation 3.3 Other Notations 4 Efficient LLM Evaluation via Control Variates 4.1 Control Variates Human annotation saving ratio. 4.2 Control Variates Evaluation
  How does the concept of human annotation saving ratio relate to the use of control variates in efficient LLM evaluation?	Accelerating Unbiased LLM Evaluation via Synthetic Feedback 1 Introduction 2 Related Work 2.1 LLM Evaluation: Metric, Benchmark and Systems 2.2 Speeding Up LLM Evaluation 2.3 Control Variates, Application, and related techniques 3 Preliminaries 3.1 LLM Evaluation 3.2 Human and Synthetic Evaluation 3.3 Other Notations 4 Efficient LLM Evaluation via Control Variates 4.1 Control Variates Human annotation saving ratio. 4.2 Control Variates Evaluation
  What are the key steps involved in the Control Variates Evaluation process as outlined in the context?	4.2 Control Variates Evaluation Synthetic annotation gathering (Line 4). Human annotation sampling (Line 5). Synthetic win rate estimation (Line 6). Control variates coefficient computation (Line 7). Win rate estimation (Line 8). (Optional) Synthetic evaluator finetuning (Line 3). Summary. 5 Experiments 5.1 Setup Synthetic evaluators. Finetuning procedure. Benchmark. 5.2 Control Variates Evaluation v.s. Human Evaluation Human annotation saving ratio on different benchmarks and synthetic evaluators. Theory matches practice.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	cosine_ndcg@10
0.3185	50	-	0.9539
0.6369	100	-	0.9826
0.9554	150	-	0.9726
1.0	157	-	0.9852
1.2739	200	-	0.9826
1.5924	250	-	0.9826
1.9108	300	-	0.9826
2.0	314	-	0.9826
2.2293	350	-	0.9752
2.5478	400	-	0.9852
2.8662	450	-	0.9852
3.0	471	-	0.9852
3.1847	500	0.3143	0.9752
3.5032	550	-	0.9752
3.8217	600	-	0.9852
4.0	628	-	0.9852
4.1401	650	-	0.9779
4.4586	700	-	0.9826
4.7771	750	-	0.9852
5.0	785	-	0.9852
5.0955	800	-	0.9852
5.4140	850	-	0.9852
5.7325	900	-	0.9826
6.0	942	-	0.9779
6.0510	950	-	0.9779
6.3694	1000	0.0878	0.9852
6.6879	1050	-	0.9779
7.0	1099	-	0.9852
7.0064	1100	-	0.9852
7.3248	1150	-	0.9852
7.6433	1200	-	0.9852
7.9618	1250	-	0.9852
8.0	1256	-	0.9852
8.2803	1300	-	0.9852
8.5987	1350	-	0.9826
8.9172	1400	-	0.9852
9.0	1413	-	0.9852
9.2357	1450	-	0.9826
9.5541	1500	0.0422	0.9826
9.8726	1550	-	0.9752
10.0	1570	-	0.9752

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 2.14.4
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}