Answers.Body: <p>Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T11:20:55.543Z
Answers.Text: GPT-2 is a decoder, not an encoder.
Answers.UserDisplayName: Green grün 绿色 vert зеленый
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-19T01:36:42.877Z
Answers.LastActivityDate: 2019-10-19T01:36:42.877Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T11:25:17.803Z
Answers.Text: It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the [Sparse Transformer](https://arxiv.org/abs/1904.10509).
Answers.UserDisplayName: Green grün 绿色 vert зеленый
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T06:11:56.817Z
Answers.LastActivityDate: 2023-04-13T06:11:56.817Z
Answers.UserDisplayName: anivana 
Body: <p>Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?</p>
  
Tags: <natural-language-processing><prediction><time-series><gpt>
Title: Is it possible to use the GPT-2 model for time-series data prediction?
UserDisplayName: xendi
ThreadText: # Is it possible to use the GPT-2 model for time-series data prediction?
  
  (Asked by: username_0 on 2019-10-18)
  
  Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
  
  **Tags:** natural-language-processing, prediction, time-series, gpt
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-19. Score: 1
  
  Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.
  
  > **Comments:**
  > **username_2**: GPT-2 is a decoder, not an encoder.  
  
  ---
  
  ### Answer by username_3 on 2023-04-13. Score: 0
  
  In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.
  
  > **Comments:**
  > **username_2**: It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the [Sparse Transformer](https://arxiv.org/abs/1904.10509).  
  
  ---
  
  