Answers.Body: <p>The embedding space of different models may be more similar than you think. Your problem reminds me a bit of blackbox adversarial attacks, where you want to e.g., find an adversarial example targeting some API to induce an output without having access to model weights.</p>
  <p>For example, <a href="https://arxiv.org/pdf/2307.15043.pdf" rel="nofollow noreferrer">Zou et al., 2023</a> and <a href="https://arxiv.org/pdf/1908.07125.pdf" rel="nofollow noreferrer">Wallace et al., 2019</a> both suggest finding adversarial examples with whitebox access to an ensemble of models (through gradient steps + discrete updates), with the goal being to induce certain types of outputs (e.g., toxic/harmful, different sentiment etc.). They then notice that these examples transfer to models that you only have API access to. This is pretty directly transferrable to your setting: take gradient steps + discrete updates (as described in the papers) to match the embeddings of an ensemble of whitebox models, then this might end up transferring to whatever blackbox API you're targeting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T19:15:37.593Z
Answers.LastActivityDate: 2024-01-19T19:15:37.593Z
Answers.UserDisplayName: Alexander Wan
Body: <p><strong>tldr;</strong> I am researching/looking for vec2text methods that work (a) without training a special model to guide the mutations, and (b) without access to the model weights.</p>
  <p>I am doing some experiments in navigating the encoded space of an encoder-only text embedding model (such as OpenAI's <a href="https://platform.openai.com/docs/guides/embeddings/use-cases" rel="nofollow noreferrer">text-embedding-ada-002</a>, but not specific to this model), with the <strong>limitation that I do not have access to the weights</strong>. I am trying to find some heuristics to &quot;travel&quot; towards a <code>target</code> embedding within the space.</p>
  <p>Note: this is about text embedding (e.g sentences, maybe paragraphs), not word embedding.</p>
  <p>Some ideas I currently have:</p>
  <ul>
  <li><p>Running an evolutionary algorithm to mutate the <code>text</code> in <code>(text, embedding)</code> pairs as individuals, and using <code>dist(embedding, target embedding)</code> as an objective.</p>
  </li>
  <li><p>At each step, locally finding &quot;directional&quot; vectors by taking the delta of two steps combined with some adjective, and associating meaning to the direction.</p>
  </li>
  <li><p>Applying a direction repeatedly if it gets me closer to the target.</p>
  </li>
  <li><p>Asking an LLM to {mutate, combine} an/two individuals' text, and also to tell me the semantic difference (giving meaning to the direction).</p>
  </li>
  <li><p>Inspiration/existing research: I am aware of the Vec2Text from <a href="https://arxiv.org/abs/2310.06816" rel="nofollow noreferrer">Text Embeddings Reveal (Almost) As Much As Text</a>, but i am specifically looking to do as well as possible <strong>without training a special model to guide the mutations</strong>.</p>
  </li>
  <li><p>Q: Is there any existing research on this sort of thing?</p>
  </li>
  <li><p>Q: Any better mutation heuristics? Other approaches entirely?</p>
  </li>
  <li><p>Q: I vaguely aware of the <a href="https://en.wikipedia.org/wiki/Manifold_hypothesis" rel="nofollow noreferrer">Manifold Hypothesis</a>, are there ways to detect if an embedding is far from the manifold? Or a way to &quot;get back&quot; to the manifold?</p>
  </li>
  </ul>
  
Tags: <embeddings><evolutionary-algorithms>
Title: Navigating encoder-only text/sentence embedding models
UserDisplayName: Realz Slaw
ThreadText: # Navigating encoder-only text/sentence embedding models
  
  (Asked by: username_0 on 2024-01-11)
  
  **tldr;** I am researching/looking for vec2text methods that work (a) without training a special model to guide the mutations, and (b) without access to the model weights.
  
  I am doing some experiments in navigating the encoded space of an encoder-only text embedding model (such as OpenAI's [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings/use-cases), but not specific to this model), with the **limitation that I do not have access to the weights**. I am trying to find some heuristics to "travel" towards a `target` embedding within the space.
  
  Note: this is about text embedding (e.g sentences, maybe paragraphs), not word embedding.
  
  Some ideas I currently have:
  
  * Running an evolutionary algorithm to mutate the `text` in `(text, embedding)` pairs as individuals, and using `dist(embedding, target embedding)` as an objective.
  * At each step, locally finding "directional" vectors by taking the delta of two steps combined with some adjective, and associating meaning to the direction.
  * Applying a direction repeatedly if it gets me closer to the target.
  * Asking an LLM to {mutate, combine} an/two individuals' text, and also to tell me the semantic difference (giving meaning to the direction).
  * Inspiration/existing research: I am aware of the Vec2Text from [Text Embeddings Reveal (Almost) As Much As Text](https://arxiv.org/abs/2310.06816), but i am specifically looking to do as well as possible **without training a special model to guide the mutations**.
  * Q: Is there any existing research on this sort of thing?
  * Q: Any better mutation heuristics? Other approaches entirely?
  * Q: I vaguely aware of the [Manifold Hypothesis](https://en.wikipedia.org/wiki/Manifold_hypothesis), are there ways to detect if an embedding is far from the manifold? Or a way to "get back" to the manifold?
  
  **Tags:** embeddings, evolutionary-algorithms
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-19. Score: 1 | Accepted answer
  
  The embedding space of different models may be more similar than you think. Your problem reminds me a bit of blackbox adversarial attacks, where you want to e.g., find an adversarial example targeting some API to induce an output without having access to model weights.
  
  For example, [Zou et al., 2023](https://arxiv.org/pdf/2307.15043.pdf) and [Wallace et al., 2019](https://arxiv.org/pdf/1908.07125.pdf) both suggest finding adversarial examples with whitebox access to an ensemble of models (through gradient steps + discrete updates), with the goal being to induce certain types of outputs (e.g., toxic/harmful, different sentiment etc.). They then notice that these examples transfer to models that you only have API access to. This is pretty directly transferrable to your setting: take gradient steps + discrete updates (as described in the papers) to match the embeddings of an ensemble of whitebox models, then this might end up transferring to whatever blackbox API you're targeting.
  
  
  ---
  
  