README.md

---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:100
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
base_model: Alibaba-NLP/gte-Qwen2-1.5B-instruct
widget:
- source_sentence: "1. What is the significance of the beginning of thinking and end\
    \ of thinking tokens in the context of recurrent depth?  \n2. How does the concept\
    \ of thinking about a single token relate to the overall sequence in the discussion?"
  sentences:
  - done by researchers at Lawrence Livermore and elsewhere um this is not this not
    you know something that product teams are using today this is this is very much
    a research grade thing that is cool and we're seeing some you know early signs
    that it's potentially quite useful um I I wanna I want to zoom in on on like just
    when people think about the the actual how of this when they think about actually
    implementing this in in maybe one of their applications so whereas in the Coconut
    space you're you're going to go and you're G to you're gonna like oh nope not
    going out into natural language space just going to sit here and chew on it chew
    on it chew on it and then I'm gonna pop out my final answer so all you get is
    final answer baby it's called a blackbox okay when we go to the recurrent depth
    piece um you said something interesting earlier when we were chatting about this
    and it was it was like I'm going to think and think and think and think and think
    and all of a sudden I know and I'm
  - chains of thought and this is where this idea of test time compute came up and
    this was a paper from Google in August last year called scaling test time compute
    you know it's basically taking that scaling paper originally and saying well now
    we have this sort of other axis to scale on and again this is the idea that we're
    anthropomorphizing a little bit but humans tend to think longer on difficult problems
    maybe we should let machines do that and when we think of test time Compu it's
    just time spent thinking you know and so if we we think about kind of how we can
    leverage this we've seen some of these things come out in recent weeks and recent
    months we talked about deep seek R1 just last week and you know this is the same
    idea it thinks before it answers and this is again just sort of the next step
    in the evolution of what we've got going on here and we saw moreover deep seek
    one generates one token at a time it's able to spend more time processing and
    it generates these thinking
  - piece um you said something interesting earlier when we were chatting about this
    and it was it was like I'm going to think and think and think and think and think
    and all of a sudden I know and I'm going to do that just with one token is sort
    of the recurrent depth paper and then I'm going to let the rest of the tokens
    stream like normal right so so there's this real interesting idea of like which
    things are you thinking about and this is where the idea of a beginning of thinking
    end of thinking token or a beginning of sequence end of sequence token comes into
    play you can think about the sequence or you can think about the thinking is this
    does this make sense um yeah okay okay we can think about the thinking right so
    we can uh we can double we can double it up uh yeah we can think about the thing
    yeah yeah okay okay so so recurrent depth in short I mean is like you think about
    a single token and then you let the sequence go like that's what I thought was
    interesting and and maybe
- source_sentence: '1. What are the different systems mentioned in the context, and
    how are they categorized?

    2. How does the concept of "Meta Meta learning" relate to the discussion in the
    context?'
  sentences:
  - right we kind of got to go a little bit more into the blackbox we gota go back
    beyond the unknown yeah it happens but it's it's it's it's the the timing is right
    and uh with companies like you know Nvidia with companies the other accelerators
    that are that are coming out they're super good at inference Gro and S all these
    other peeps right uh we're getting real fast at inference and so the spending
    that time you know becomes less and less impactful to the user experience but
    more importantly uh you know we have a lot of applications LMS aren't good for
    yet where we don't care about response latency like research like uh PhD level
    math where it's like it doesn't matter if it takes a day yeah because that means
    it didn't take some some other person a day right like that's the that's the the
    we're at this time the models are capable enough that we can think about problems
    that we can't just do ourselves faster it the whole the whole you know ecosystem
    is set up for this to be the right
  - perceptron artificial neural networks and single neurons to many neurons and so
    on but it really really got going with deep learning and we we saw we train bigger
    and bigger models we get better and better output and results this has been known
    for years and it's it's known even today I mean this is from the CEO of anthropic
    Dario amade and you know we want to think about the the the place that we are
    based on where we've been want to put this in context here so we go from pre-training
    which is the thing that sort of takes a long time it's the it's the show goth
    it's got all the possibilities we can sort of think of this as months many many
    tens of thousands of gpus maybe more these days and as we saw at nurs this past
    uh you know year Ilia suit noted that pre-training as we know it will end now
    we've seen that there's been a lot of focus subsequently especially you know out
    in public on posttraining and on this idea of rhf and RL we talked about this
    in our recent reasoning model
  - we've got we've got sort of this idea may we could call it system one is GPT 40
    mini and then we could say Okay reasoning model that's spitting out tokens is
    like is like system two and then maybe like we've got this sort of system three
    that's in in in latent space and then we've got this like system four that's like
    that's like you know On Any Given specific type of of of thing I might want to
    extra think about in latent space I might go deeper on it so so there's just this
    real sort of very interesting abstraction Ed meta thinking at play here and um
    and it gets back to me to sort of the idea of like kind of Meta Meta learning
    I mean they they really they really did away with our ability to use that term
    and in the gpt3 paper so um we're at the end of sort of the words that we that
    we can concretely say we know what to do but we know what to do with the code
    so I want to go ahead and just quickly introduce this to you uh whiz we're going
    to do a quick demo here and we're going to
- source_sentence: '1. What does the term "tokenless" refer to in the context of scaling
    and model architecture?

    2. How does the looping process contribute to generating responses without resolving
    back to tokens?'
  sentences:
  - allow us to scil uh that seems a little a little weird to me I'm not it's not
    very intuitive what's your intuition behind this yeah so I the word tokenless
    is a is a is a fun one I would say like the idea is we don't need to resolve back
    to tokens to scale because we can just keep looping around before we get to tokens
    right really anything that allows us to keep looping around until we get to The
    Final Answer uh is is g to allow us to scale right because we can say well Loop
    more Loop more Loop more right like uh W with with uh you know with say like generating
    a response right if I generate for five tokens versus if I generate for 70,000
    tokens right like we we have this idea of a scaling axis on the on the inference
    side this is just kind of shoving that inside the uh the model architecture as
    opposed to allowing it to resolve to token space but the idea is we're still adding
    information at every step at every stage right that we do a a loop as it were
    so we're getting a better and
  - perceptron artificial neural networks and single neurons to many neurons and so
    on but it really really got going with deep learning and we we saw we train bigger
    and bigger models we get better and better output and results this has been known
    for years and it's it's known even today I mean this is from the CEO of anthropic
    Dario amade and you know we want to think about the the the place that we are
    based on where we've been want to put this in context here so we go from pre-training
    which is the thing that sort of takes a long time it's the it's the show goth
    it's got all the possibilities we can sort of think of this as months many many
    tens of thousands of gpus maybe more these days and as we saw at nurs this past
    uh you know year Ilia suit noted that pre-training as we know it will end now
    we've seen that there's been a lot of focus subsequently especially you know out
    in public on posttraining and on this idea of rhf and RL we talked about this
    in our recent reasoning model
  - allow us to scil uh that seems a little a little weird to me I'm not it's not
    very intuitive what's your intuition behind this yeah so I the word tokenless
    is a is a is a fun one I would say like the idea is we don't need to resolve back
    to tokens to scale because we can just keep looping around before we get to tokens
    right really anything that allows us to keep looping around until we get to The
    Final Answer uh is is g to allow us to scale right because we can say well Loop
    more Loop more Loop more right like uh W with with uh you know with say like generating
    a response right if I generate for five tokens versus if I generate for 70,000
    tokens right like we we have this idea of a scaling axis on the on the inference
    side this is just kind of shoving that inside the uh the model architecture as
    opposed to allowing it to resolve to token space but the idea is we're still adding
    information at every step at every stage right that we do a a loop as it were
    so we're getting a better and
- source_sentence: '1. What is the relationship between latent space and natural language
    in the context of a Transformer architecture?

    2. How does the GPT style architecture process a sequence to predict the next
    token?'
  sentences:
  - piece um you said something interesting earlier when we were chatting about this
    and it was it was like I'm going to think and think and think and think and think
    and all of a sudden I know and I'm going to do that just with one token is sort
    of the recurrent depth paper and then I'm going to let the rest of the tokens
    stream like normal right so so there's this real interesting idea of like which
    things are you thinking about and this is where the idea of a beginning of thinking
    end of thinking token or a beginning of sequence end of sequence token comes into
    play you can think about the sequence or you can think about the thinking is this
    does this make sense um yeah okay okay we can think about the thinking right so
    we can uh we can double we can double it up uh yeah we can think about the thing
    yeah yeah okay okay so so recurrent depth in short I mean is like you think about
    a single token and then you let the sequence go like that's what I thought was
    interesting and and maybe
  - it's kind of funny in a logical way if you look up logic it uses the word reason
    and there we are caught in a loop but reasoning is about thinking latent space
    is about using a representation of our data that sort of captures the essential
    features of it we can think of latent space as embedding space or the space of
    math and numbers in other words it's just not the space of words and natural language
    let's think about how this manifests in a Transformer architecture here I'm showing
    a GPT style architecture from the gpt2 paper what we want to think about is we
    want to put a sequence in and we want to get some next token prediction out when
    we put the sequence in we're in the space of natural language when we get the
    next token out we're in the space of natural language betwix in between we're
    going to be in latent space we're going to be in embedding space we're going to
    be in the space where we can do math and stuff and importantly we can kind of
    think that we're putting in this big
  - architecture is built upon a latent depth recurrent block that is run for a randomly
    sampled number of iterations during training I can't see it let's see what they
    gave us in the paper they gave us this bad boy personally not a fan of this diagram
    whiz thinks it's totally fine and it makes perfect sense let's see if we can break
    it down for you guys here a visualization of the architecture we have the Prelude
    block we have the recurrent block recurrent block recurrent block and then this
    Koda so each block consists of a number of su layers okay the blue Prelude block
    embeds the input into latent space all right where the green shared recurrent
    block is a block of layers that is repeated to compute the final latent state
    which is decoded by the layers of the red Coda block back to our gp2 architecture
    diagram let's think about how we're still kind of doing this loop back we're still
    doing this reasoning in in space and now let's label the Prelude the recurrent
    block and the Koda we
- source_sentence: '1. What is meant by the terms "hidden state," "latent space,"
    and "embedding space" in the context of reasoning models?

    2. How do the last hidden states function as input embeddings in a typical Chain
    of Thought reasoning model?'
  sentences:
  - the next step in the evolution of what we've got going on here and we saw moreover
    deep seek one generates one token at a time it's able to spend more time processing
    and it generates these thinking tokens that explain its Chain of Thought So we're
    generating tokens during our chains of thought and that makes them explainable
    very cool all right I want to bring whiz back up for just a moment here okay just
    to be super clear reasoning and test time compute do you think these are sort
    of you know triple equal sign or or how would you how would you say you know generally
    we're talking about the same thing but they're not the same thing yeah they're
    they're this so so okay they're not literally the same thing of course but they're
    also pretty much the same thing in in how we talk about it in 2025 today that's
    right that's right so so reasoning is some right now because our models are System
    One machines right this is the this is the they're not reasoners they're they're
    uh they're they're
  - impact of this kind of approach on test time compute scaling some of the working
    hypotheses and some of the things people are interested in in looking out there
    on the llm edge for as we continue to see the field progress I want to demonstrate
    both approaches and check out the new coconut Library as well so how we're going
    to go through this is we're going to essentially introduce this idea of reasoning
    and latent space then we're going to talk about the scaling part of this before
    we dig into the specific approaches and we get the demo on both approaches by
    the end so it should be a lot of fun today let's go ahead and dig in reasoning
    in latent space let's root ourselves first in some definitions when we talk about
    reasoning we're talking about the action of thinking about something and it's
    kind of funny in a logical way if you look up logic it uses the word reason and
    there we are caught in a loop but reasoning is about thinking latent space is
    about using a representation of our
  - of the reasoning State when we say hidden state or latent space or embedding space
    or this sort of space of math and computation we're talking about the same space
    of course the the exact state of the space changes depending on where we are in
    the Transformer but let's take a look at the image from the paper in a typical
    Chain of Thought reasoning model we're going to ask a question we're going to
    generate some tokens and we're going to think kind of out loud we're we're going
    to let the chains of thought flow when you click into the Chain of Thought on
    01 as we've seen before you can see sort of in the side panel the the steps it's
    thinking through now conversely to actually thinking out loud we have here that
    the last hidden states are used as input embeddings okay well what does this mean
    well it let's go back to our gpt2 style diagram and think about this the input
    embeddings here are where we're essentially looping back to so what we do is we
    kind of loop back before we generate
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
model-index:
- name: SentenceTransformer based on Alibaba-NLP/gte-Qwen2-1.5B-instruct
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: Unknown
      type: unknown
    metrics:
    - type: cosine_accuracy@1
      value: 0.875
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 1.0
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 1.0
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 1.0
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.875
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3333333333333333
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.2
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.1
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.875
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 1.0
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 1.0
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 1.0
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9538662191964322
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9375
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9375
      name: Cosine Map@100
---

# SentenceTransformer based on Alibaba-NLP/gte-Qwen2-1.5B-instruct

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct). It maps sentences & paragraphs to a 1536-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) <!-- at revision 0d2ad8e1ac654a2b626e62154778a70868141208 -->
- **Maximum Sequence Length:** 32768 tokens
- **Output Dimensionality:** 1536 dimensions
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kenrogers/gte-ft-yt-2")
# Run inference
sentences = [
    '1. What is meant by the terms "hidden state," "latent space," and "embedding space" in the context of reasoning models?\n2. How do the last hidden states function as input embeddings in a typical Chain of Thought reasoning model?',
    "of the reasoning State when we say hidden state or latent space or embedding space or this sort of space of math and computation we're talking about the same space of course the the exact state of the space changes depending on where we are in the Transformer but let's take a look at the image from the paper in a typical Chain of Thought reasoning model we're going to ask a question we're going to generate some tokens and we're going to think kind of out loud we're we're going to let the chains of thought flow when you click into the Chain of Thought on 01 as we've seen before you can see sort of in the side panel the the steps it's thinking through now conversely to actually thinking out loud we have here that the last hidden states are used as input embeddings okay well what does this mean well it let's go back to our gpt2 style diagram and think about this the input embeddings here are where we're essentially looping back to so what we do is we kind of loop back before we generate",
    "impact of this kind of approach on test time compute scaling some of the working hypotheses and some of the things people are interested in in looking out there on the llm edge for as we continue to see the field progress I want to demonstrate both approaches and check out the new coconut Library as well so how we're going to go through this is we're going to essentially introduce this idea of reasoning and latent space then we're going to talk about the scaling part of this before we dig into the specific approaches and we get the demo on both approaches by the end so it should be a lot of fun today let's go ahead and dig in reasoning in latent space let's root ourselves first in some definitions when we talk about reasoning we're talking about the action of thinking about something and it's kind of funny in a logical way if you look up logic it uses the word reason and there we are caught in a loop but reasoning is about thinking latent space is about using a representation of our",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1536]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Information Retrieval

* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.875      |
| cosine_accuracy@3   | 1.0        |
| cosine_accuracy@5   | 1.0        |
| cosine_accuracy@10  | 1.0        |
| cosine_precision@1  | 0.875      |
| cosine_precision@3  | 0.3333     |
| cosine_precision@5  | 0.2        |
| cosine_precision@10 | 0.1        |
| cosine_recall@1     | 0.875      |
| cosine_recall@3     | 1.0        |
| cosine_recall@5     | 1.0        |
| cosine_recall@10    | 1.0        |
| **cosine_ndcg@10**  | **0.9539** |
| cosine_mrr@10       | 0.9375     |
| cosine_map@100      | 0.9375     |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### Unnamed Dataset

* Size: 100 training samples
* Columns: <code>sentence_0</code> and <code>sentence_1</code>
* Approximate statistics based on the first 100 samples:
  |         | sentence_0                                                                         | sentence_1                                                                            |
  |:--------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|
  | type    | string                                                                             | string                                                                                |
  | details | <ul><li>min: 30 tokens</li><li>mean: 41.05 tokens</li><li>max: 60 tokens</li></ul> | <ul><li>min: 180 tokens</li><li>mean: 208.98 tokens</li><li>max: 231 tokens</li></ul> |
* Samples:
  | sentence_0                                                                                                                                                                                                                                   | sentence_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
  |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  | <code>1. What are the two big ideas aimed at scaling the power of LLMs during inference mentioned in the context?  <br>2. How does the concept of reasoning in latent space relate to the efficiency of computation during inference?</code> | <code>okay whiz we're talking about reasoning in latent space today is that the same as test time compute yeah that's right nice nice okay and we've got two big ideas to cover that are aimed at scaling the power of llms during inference is that right that yeah that's right so we have we have two you know latent space methods uh we have our continuous Chain of Thought or coconut right and then we have our more more directly more uh you know uh budget forcing recurrent depth uh model yes man that's a lot so when we look across both of those there appears to be a pretty simple explanation it's almost like uh you know when we when we're in that sort of thinking space of computation we don't have to do the thinky thinky in words and that's better maybe even it will allow us to find a new scaling axis is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not</code> |
  | <code>1. What are the two big ideas aimed at scaling the power of LLMs during inference mentioned in the context?  <br>2. How does the concept of reasoning in latent space relate to the efficiency of computation during inference?</code> | <code>okay whiz we're talking about reasoning in latent space today is that the same as test time compute yeah that's right nice nice okay and we've got two big ideas to cover that are aimed at scaling the power of llms during inference is that right that yeah that's right so we have we have two you know latent space methods uh we have our continuous Chain of Thought or coconut right and then we have our more more directly more uh you know uh budget forcing recurrent depth uh model yes man that's a lot so when we look across both of those there appears to be a pretty simple explanation it's almost like uh you know when we when we're in that sort of thinking space of computation we don't have to do the thinky thinky in words and that's better maybe even it will allow us to find a new scaling axis is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not</code> |
  | <code>1. What is the significance of staying in the "mind Palace" of the Transformer instead of resolving back to token space?  <br>2. What are the key concepts that need to be covered before demonstrating large reasoning models?</code> | <code>is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not just like for a second right not automatically resolving back to token space but kind of staying in this very like uh you know in in the mind Palace of the of the Transformer without having to write down the words yes okay okay okay so basically scaling is dead Long Live scaling something like that yeah scaling has died uh we should scale yeah all right all right all right well I'm pumped for the demos today we're going to see some thinking in latent space let's cover all the Concepts we need to get there we'll get you back in for some discussions along the way because this one's pretty meta thanks whiz all right guys we are gonna rock out on large reasoning models today while we were originally going to just cover chain of continuous thought or coconut we saw a paper come out a couple</code>     |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
  ```json
  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  ```

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: steps
- `per_device_train_batch_size`: 10
- `per_device_eval_batch_size`: 10
- `num_train_epochs`: 10
- `multi_dataset_batch_sampler`: round_robin

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 10
- `per_device_eval_batch_size`: 10
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 5e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1
- `num_train_epochs`: 10
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.0
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: None
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: round_robin

</details>

### Training Logs
| Epoch | Step | cosine_ndcg@10 |
|:-----:|:----:|:--------------:|
| 1.0   | 10   | 0.9539         |
| 2.0   | 20   | 0.9077         |
| 3.0   | 30   | 0.9539         |
| 4.0   | 40   | 0.9539         |
| 5.0   | 50   | 0.9539         |
| 6.0   | 60   | 0.9539         |
| 7.0   | 70   | 0.9539         |
| 8.0   | 80   | 0.9539         |
| 9.0   | 90   | 0.9539         |
| 10.0  | 100  | 0.9539         |


### Framework Versions
- Python: 3.11.11
- Sentence Transformers: 3.4.1
- Transformers: 4.48.3
- PyTorch: 2.5.1+cu124
- Accelerate: 1.3.0
- Datasets: 3.3.2
- Tokenizers: 0.21.0

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->