Answers.Body: <p>Building transformer models, in general, for <em>arbitrary</em> context retrieval is difficult because the architecture and training simply aren't designed to handle this. This is even more of an issue if you want to use the model to embed documents, as you typically need to do some specialized training to get meaningful embeddings.</p>
  <p>There has, however, been work on long-context embedding models more generally: e.g., <a href="https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval" rel="nofollow noreferrer">M2-BERT with a maximum of 32k tokens</a>. i.e., it can support documents with length &lt;= 32k. This, however, isn't a transformer model. Comparing to other models in <a href="https://arxiv.org/pdf/2402.07440v2.pdf" rel="nofollow noreferrer">their paper</a>, transformer-based embedding methods only seem to have a maximum context length of 8k.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T23:33:56.733Z
Answers.Text: Ok, so is M2-BERT a state space model, kind of like RWKV and Mamba?
Answers.UserDisplayName: HelloGoodbye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-07T02:00:03.470Z
Answers.Text: I don't think RWKV is a state space model, but it would be more similar to Mamba.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T03:49:45.713Z
Answers.LastActivityDate: 2024-03-04T03:49:45.713Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>One way you could do it is by using <strong>SentenceTransformers</strong>.</p>
  <blockquote>
  <p>SentenceTransformers is a Python framework for state-of-the-art
  sentence, text and image embeddings. The initial work is described in
  our paper <a href="https://arxiv.org/abs/1908.10084" rel="nofollow noreferrer">Sentence-BERT: Sentence Embeddings using Siamese
  BERT-Networks.</a></p>
  </blockquote>
  <p>To install it via pip</p>
  <pre><code>pip install -U sentence-transformers
  </code></pre>
  <p>To generate sentence embedding</p>
  <pre><code>from sentence_transformers import SentenceTransformer
  
  # We are using &quot;paraphrase-MiniLM-L6-v2&quot; model here, You can find list of model [here][2]
  model = SentenceTransformer('paraphrase-MiniLM-L6-v2')
  
  # Our sentences we like to encode
  sentences = ['This framework generates embeddings for each input sentence',
      'Sentences are passed as a list of string.',
      'The quick brown fox jumps over the lazy dog.']
  
  # Sentences are encoded by calling model.encode()
  embeddings = model.encode(sentences)
  
  #Print the embeddings
  for sentence, embedding in zip(sentences, embeddings):
      print(&quot;Sentence:&quot;, sentence)
      print(&quot;Embedding:&quot;, embedding)
      print(&quot;&quot;)
  </code></pre>
  <p>But remember, SentenceTransformers models have an input limit as well, 512 tokens usually. If your text is more than that, then it might not be a suitable method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-11T21:23:03.287Z
Answers.Text: Thanks for you answer. Well, if it has an input limit, the input isn’t truly of arbitrary length (in practice I guess the input also becomes of fixed length, and that you just use a padding token to fill it out after the text has ended). I was wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.
Answers.UserDisplayName: HelloGoodbye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T04:34:44.920Z
Answers.Text: Not till today.
Answers.UserDisplayName: Shaida Muhammad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-10T04:32:11.323Z
Answers.LastActivityDate: 2022-02-10T04:32:11.323Z
Answers.UserDisplayName: Shaida Muhammad
Body: <p><a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT</a> encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of the input length of the text, which makes it more cumbersome to use as input to downstream neural networks that take only fixed-size inputs.</p>
  <p>Are there any transformer-based neural network architectures that can encode a piece of text into a fixed-size feature vector more suitable for downstream tasks?</p>
  <p><strong>Edit:</strong> To illustrate my question, I’m wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-18T13:12:10.913Z
Comments.Text: [This](https://ai.stackexchange.com/questions/22957/how-can-transformers-handle-arbitrary-length-input?rq=1#comment35118_22960) might help and partially answers your question. You can only try to reduce the size of the obtained state with some convolution, but I don't think this is being done yet.
Comments.UserDisplayName: N. Kiefer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-21T16:14:16.883Z
Comments.Text: BERT does provide a fixed-size output. The encoding of the special `[CLS]` token, which is always prepended to every input example, is meant to encode the entire sentence. Since it is a token, its encoding is always a fixed-length vector of length _H_, e.g. H=768 for BERT-Base. Specifically, the `[CLS]` encoding is passed through a "pooling layer", which is just a _HxH_ fully-connected layer, with linear activation. This is called `pooled_output` in the [TF Hub module](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2).
Comments.UserDisplayName: primussucks
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-30T15:43:19.797Z
Comments.Text: As mentioned, use the CLS token. Else you could just define some max_length, and pad to it when its to low and then use the mean of the tokens. But, look at sentence-bert for e.g https://www.sbert.net/#usage - it generates fixed sized sentence embeddings with varying input sizes :)
Comments.UserDisplayName: Isbister
Tags: <natural-language-processing><reference-request><autoencoders><transformer><bert>
Title: Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
UserDisplayName: HelloGoodbye
ThreadText: # Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
  
  (Asked by: username_0 on 2020-09-15)
  
  [BERT](https://arxiv.org/abs/1810.04805) encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of the input length of the text, which makes it more cumbersome to use as input to downstream neural networks that take only fixed-size inputs.
  
  Are there any transformer-based neural network architectures that can encode a piece of text into a fixed-size feature vector more suitable for downstream tasks?
  
  **Edit:** To illustrate my question, I’m wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.
  
  **Tags:** natural-language-processing, reference-request, autoencoders, transformer, bert
  > **Comments:**
  > **username_1**: [This](https://ai.stackexchange.com/questions/22957/how-can-transformers-handle-arbitrary-length-input?rq=1#comment35118_22960) might help and partially answers your question. You can only try to reduce the size of the obtained state with some convolution, but I don't think this is being done yet.  
  > **username_2**: BERT does provide a fixed-size output. The encoding of the special `[CLS]` token, which is always prepended to every input example, is meant to encode the entire sentence. Since it is a token, its encoding is always a fixed-length vector of length _H_, e.g. H=768 for BERT-Base. Specifically, the `[CLS]` encoding is passed through a "pooling layer", which is just a _HxH_ fully-connected layer, with linear activation. This is called `pooled_output` in the [TF Hub module](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2).  
  > **username_3**: As mentioned, use the CLS token. Else you could just define some max_length, and pad to it when its to low and then use the mean of the tokens. But, look at sentence-bert for e.g https://www.sbert.net/#usage - it generates fixed sized sentence embeddings with varying input sizes :)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2024-03-04. Score: 0
  
  Building transformer models, in general, for *arbitrary* context retrieval is difficult because the architecture and training simply aren't designed to handle this. This is even more of an issue if you want to use the model to embed documents, as you typically need to do some specialized training to get meaningful embeddings.
  
  There has, however, been work on long-context embedding models more generally: e.g., [M2-BERT with a maximum of 32k tokens](https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval). i.e., it can support documents with length <= 32k. This, however, isn't a transformer model. Comparing to other models in [their paper](https://arxiv.org/pdf/2402.07440v2.pdf), transformer-based embedding methods only seem to have a maximum context length of 8k.
  
  > **Comments:**
  > **username_0**: Ok, so is M2-BERT a state space model, kind of like RWKV and Mamba?  
  > **username_4**: I don't think RWKV is a state space model, but it would be more similar to Mamba.  
  
  ---
  
  ### Answer by username_5 on 2022-02-10. Score: 0
  
  One way you could do it is by using SentenceTransformers.
  
  SentenceTransformers is a Python framework for state-of-the-art
  sentence, text and image embeddings. The initial work is described in
  our paper Sentence-BERT: Sentence Embeddings using Siamese
  BERT-Networks.
  
  To install it via pip
  pip install -U sentence-transformers
  
  To generate sentence embedding
  from sentence_transformers import SentenceTransformer
  
  # We are using "paraphrase-MiniLM-L6-v2" model here, You can find list of model [here][2]
  model = SentenceTransformer('paraphrase-MiniLM-L6-v2')
  
  # Our sentences we like to encode
  sentences = ['This framework generates embeddings for each input sentence',
      'Sentences are passed as a list of string.',
      'The quick brown fox jumps over the lazy dog.']
  
  # Sentences are encoded by calling model.encode()
  embeddings = model.encode(sentences)
  
  #Print the embeddings
  for sentence, embedding in zip(sentences, embeddings):
      print("Sentence:", sentence)
      print("Embedding:", embedding)
      print("")
  
  But remember, SentenceTransformers models have an input limit as well, 512 tokens usually. If your text is more than that, then it might not be a suitable method.
  
  > **Comments:**
  > **username_0**: Thanks for you answer. Well, if it has an input limit, the input isn’t truly of arbitrary length (in practice I guess the input also becomes of fixed length, and that you just use a padding token to fill it out after the text has ended). I was wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.  
  > **username_5**: Not till today.  
  
  ---
  
  