Answers.Body: <p>Great news -- transformers do this already. The way transformers generate discrete tokens is by using a linear layer to do classification over the vocabulary given the transformer outputs. The transformer outputs are just dense vectors in the same shape as the input token embeddings (i.e., of shape = <code>(batch_size, seq_len, embed_dim)</code>).</p>
  <p>You could train the model with something like MSE loss. Transformers don't necessarily have to predict discrete tokens (e.g., <a href="https://arxiv.org/abs/2111.06377" rel="nofollow noreferrer">the MAE paper</a> use a transformer to predict pixel values directly, and trains with MSE). Cosine-distance could also work as a loss function for embedding prediction, as used in <a href="https://arxiv.org/pdf/2206.08081.pdf" rel="nofollow noreferrer">this paper which has a transformer model that produces an embedding output</a>.</p>
  <p>However, it's important to note that it's common for people to convert the data into some discrete format first. For example, for applications like image or audio generation, you often have a VQ-VAE create a &quot;code-book&quot; of embeddings, and you'd use a transformer to model those codes as a discrete &quot;vocabulary&quot;. See <a href="https://arxiv.org/pdf/2102.12092.pdf" rel="nofollow noreferrer">this paper</a> for an example. Finally, models like imageGPT instead quantize color values to create a discrete &quot;vocabulary&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T05:05:37.207Z
Answers.LastActivityDate: 2023-07-24T08:56:10.240Z
Answers.LastEditDate: 2023-07-24T08:56:10.240Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens?</p>
  <p>For example, if I were to provide an input of a stream of audio embeddings in the format (batch_size, seq_len, embed_dim), would it theoretically be model be able to train a transformer to predict the next audio embedding in the sequence, using a linear layer instead of creating token embeddings?</p>
  <p>If this is possible, what loss function would best be used in this scenario? (for non-discrete data)</p>
  
Tags: <machine-learning><transformer><embeddings><sequence-modeling>
Title: Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?
UserDisplayName: Theo Coombes
ThreadText: # Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?
  
  (Asked by: username_0 on 2023-07-24)
  
  Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens?
  
  For example, if I were to provide an input of a stream of audio embeddings in the format (batch\_size, seq\_len, embed\_dim), would it theoretically be model be able to train a transformer to predict the next audio embedding in the sequence, using a linear layer instead of creating token embeddings?
  
  If this is possible, what loss function would best be used in this scenario? (for non-discrete data)
  
  **Tags:** machine-learning, transformer, embeddings, sequence-modeling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-24. Score: 1
  
  Great news -- transformers do this already. The way transformers generate discrete tokens is by using a linear layer to do classification over the vocabulary given the transformer outputs. The transformer outputs are just dense vectors in the same shape as the input token embeddings (i.e., of shape = `(batch_size, seq_len, embed_dim)`).
  
  You could train the model with something like MSE loss. Transformers don't necessarily have to predict discrete tokens (e.g., [the MAE paper](https://arxiv.org/abs/2111.06377) use a transformer to predict pixel values directly, and trains with MSE). Cosine-distance could also work as a loss function for embedding prediction, as used in [this paper which has a transformer model that produces an embedding output](https://arxiv.org/pdf/2206.08081.pdf).
  
  However, it's important to note that it's common for people to convert the data into some discrete format first. For example, for applications like image or audio generation, you often have a VQ-VAE create a "code-book" of embeddings, and you'd use a transformer to model those codes as a discrete "vocabulary". See [this paper](https://arxiv.org/pdf/2102.12092.pdf) for an example. Finally, models like imageGPT instead quantize color values to create a discrete "vocabulary".
  
  
  ---
  
  