Answers.Body: <p>When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a <span class="math-container">$n^2$</span> matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.</p>
  <p>OpenAI did some related research, as the followings.</p>
  <p><a href="https://openai.com/blog/sparse-transformer/" rel="nofollow noreferrer">Generative Modeling with Sparse Transformers</a>:
  In the paper, transformers with sparse attention are applied to image and waveform.</p>
  <p><a href="https://openai.com/blog/image-gpt/" rel="nofollow noreferrer">ImageGPT</a>:
  A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T15:36:39.190Z
Answers.Text: You may also want to mention the _vision transformer_.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-02T16:11:35.317Z
Answers.LastActivityDate: 2020-08-02T16:11:35.317Z
Answers.UserDisplayName: tsu
Body: <p>Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other problems)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-25T07:51:46.857Z
Comments.Text: Image generation with transformers instead of GANs is a thing.
Comments.UserDisplayName: thepacker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-25T17:02:09.713Z
Comments.Text: do you have a paper or a reference?
Comments.UserDisplayName: Schach21
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-25T17:37:04.510Z
Comments.Text: https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf see also the completions section at their website.  https://openai.com/blog/image-gpt/
Comments.UserDisplayName: thepacker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-03T08:27:32.890Z
Comments.Text: I've recently start seeing transformer-based models being used for traffic forecasting. Here is an example: https://arxiv.org/abs/2001.02908
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-30T15:35:34.757Z
Comments.Text: @razvanc92 Although this is an old question, you can provide a formal answer below. Comments are meant to be temporary.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-30T15:36:06.503Z
Comments.Text: @thepacker As I told razvanc92, you can provide a formal answer below.
Comments.UserDisplayName: nbro
Tags: <deep-learning><natural-language-processing><applications><sequence-modeling><transformer>
Title: Do transformers have success in other domains different than NLP?
UserDisplayName: Schach21
ThreadText: # Do transformers have success in other domains different than NLP?
  
  (Asked by: username_0 on 2020-06-24)
  
  Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other problems)?
  
  **Tags:** deep-learning, natural-language-processing, applications, sequence-modeling, transformer
  
  > **Comments:**
  > **username_1**: Image generation with transformers instead of GANs is a thing.  
  > **username_0**: do you have a paper or a reference?  
  > **username_1**: https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf see also the completions section at their website.  https://openai.com/blog/image-gpt/  
  > **username_2**: I've recently start seeing transformer-based models being used for traffic forecasting. Here is an example: https://arxiv.org/abs/2001.02908  
  > **username_3**: @username_2 Although this is an old question, you can provide a formal answer below. Comments are meant to be temporary.  
  > **username_3**: @username_1 As I told username_2, you can provide a formal answer below.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-08-02. Score: 1 | Accepted answer
  
  When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.
  
  OpenAI did some related research, as the followings.
  
  [Generative Modeling with Sparse Transformers](https://openai.com/blog/sparse-transformer/):
  In the paper, transformers with sparse attention are applied to image and waveform.
  
  [ImageGPT](https://openai.com/blog/image-gpt/):
  A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)
  
  > **Comments:**
  > **username_3**: You may also want to mention the _vision transformer_.  
  
  ---
  
  