<p>Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.</p>

CC BY-SA 4.0
2023-04-13T11:20:55.543Z
GPT-2 is a decoder, not an encoder.
Green grün 绿色 vert зеленый
CC BY-SA 4.0
2019-10-19T01:36:42.877Z
2019-10-19T01:36:42.877Z
mshlis
<p>In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.</p>

CC BY-SA 4.0
2023-04-13T11:25:17.803Z
It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the [Sparse Transformer](https://arxiv.org/abs/1904.10509).
Green grün 绿色 vert зеленый
CC BY-SA 4.0
2023-04-13T06:11:56.817Z
2023-04-13T06:11:56.817Z
anivana 
<p>Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?</p>

<natural-language-processing><prediction><time-series><gpt>
Is it possible to use the GPT-2 model for time-series data prediction?
xendi
# Is it possible to use the GPT-2 model for time-series data prediction?

(Asked by: username_0 on 2019-10-18)

Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?

**Tags:** natural-language-processing, prediction, time-series, gpt


---

## 2 answers

### Answer by username_1 on 2019-10-19. Score: 1

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.

> **Comments:**
> **username_2**: GPT-2 is a decoder, not an encoder.  

---

### Answer by username_3 on 2023-04-13. Score: 0

In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.

> **Comments:**
> **username_2**: It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the [Sparse Transformer](https://arxiv.org/abs/1904.10509).  

---

