Answers.Body: <p>If your data are 2D + time then you might want to use something like <a href="https://arxiv.org/abs/1506.04214" rel="nofollow noreferrer">ConvLSTM</a>.</p>
  <p>If you only care about 1D + time then you don't need to add CNN to LSTM you only use one or the other. In terms of pros and cons have a look at <a href="https://arxiv.org/abs/1803.01271" rel="nofollow noreferrer">this empirical study</a> on how dilated convolutions compare to LSTMs for modeling sequential data. If you're also interested in the more theoretical aspects, <a href="https://arxiv.org/abs/1810.06682" rel="nofollow noreferrer">this paper</a> shows how temporal convolutional networks are related to truncated RNNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-13T11:52:42.793Z
Answers.LastActivityDate: 2022-06-13T11:52:42.793Z
Answers.UserDisplayName: sfotiadis
Answers.Body: <p>A convolution layer can extract local patterns from its input (from the previous layer). Neighborhood size of the extracted pattern is relative to the kernel size. As we go deeper in the network the reception field of the layers increase. Accordingly, layers can recognize patters from larger neighborhoods of the input data. However, this increase in the reception field is very gradual. So, to understand patterns concerning large neighborhoods of input data, you need very deep networks. To ease this requirement, you can use dilated convolution layers, as in [1,2]. Hence, with shallower networks u can have reception fields as large as the whole input sequence. notice that many time series problems require patterns concerning the whole input sequence. <br/>
  using LSTM layers in combination with convolution layers is also beneficial in many cases. in this scenario, several blocks of convolution layers first extract local hierarchical patterns. So the input sequence is represented by a reach collection of local patterns. Then, this patterns are feed to an LSTM to understand dependencies related to whole sequence. also, notice that an lstm layer generally requires more parameters to learn. <br/>
  [1] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., &amp; Muller, P. A. (2019). Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4), 917-963.<br/>
  [2] Wang, Z., Yan, W., &amp; Oates, T. (2017, May). Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) (pp. 1578-1585). IEEE.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-10T13:53:12.310Z
Answers.LastActivityDate: 2022-11-10T13:53:12.310Z
Answers.UserDisplayName: Reza_va
Body: <p>I am working with simulated sequential data and the goal is to forecast that data. Long-short-term-memory (LSTM) is one of the most advanced models to forecast time series according to this <a href="https://ai.stackexchange.com/questions/27312/advantages-of-cnn-vs-lstm-for-sequence-data-like-text-or-log-files">post</a>. I can imagine that it is a good model because of the memory-cells they use which are useful when learning of the past.</p>
  <p>This <a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-022-00599-y" rel="nofollow noreferrer">paper</a> discussed the use of CNN in time-series analysis. It says:</p>
  <blockquote>
  <p>CNN is suitable for forecasting time-series because it offers dilated
  convolutions, in which filters can be used to compute dilations
  between cells. The size of the space between each cell allows the
  neural network to understand better the relationships between the
  different observations in the time-series [14].</p>
  </blockquote>
  <p>It even outperformed LSTM:</p>
  <blockquote>
  <p>A specific architecture of CNN, WaveNet, outperformed LSTM and the
  other methods in forecasting financial time-series [16].</p>
  </blockquote>
  <p>I see more and more posts about the usage of CNN in combination with LSTM, but I can't find any information about the advantages and disadvantages of using these in combination.</p>
  <p>This post (<a href="https://ai.stackexchange.com/questions/27312/advantages-of-cnn-vs-lstm-for-sequence-data-like-text-or-log-files">Advantages of CNN vs. LSTM for sequence data like text or log-files</a>), it is asked about the advantages of CNN vs. LSTM. But I would like to know the advantages and disadvantages of adding CNN to LSTM for forecasting univariate sequential data? Or should you use one of the two algorithms?</p>
  
Tags: <neural-networks><convolutional-neural-networks><long-short-term-memory>
Title: What is the advantage of adding CNN to LSTM for forecasting sequential data?
UserDisplayName: Quinten
ThreadText: # What is the advantage of adding CNN to LSTM for forecasting sequential data?
  
  (Asked by: username_0 on 2022-06-11)
  
  I am working with simulated sequential data and the goal is to forecast that data. Long-short-term-memory (LSTM) is one of the most advanced models to forecast time series according to this [post](https://ai.stackexchange.com/questions/27312/advantages-of-cnn-vs-lstm-for-sequence-data-like-text-or-log-files). I can imagine that it is a good model because of the memory-cells they use which are useful when learning of the past.
  
  This [paper](https://journalofbigdata.springeropen.com/articles/10.1186/s40537-022-00599-y) discussed the use of CNN in time-series analysis. It says:
  
  > CNN is suitable for forecasting time-series because it offers dilated
  > convolutions, in which filters can be used to compute dilations
  > between cells. The size of the space between each cell allows the
  > neural network to understand better the relationships between the
  > different observations in the time-series [14].
  
  It even outperformed LSTM:
  
  > A specific architecture of CNN, WaveNet, outperformed LSTM and the
  > other methods in forecasting financial time-series [16].
  
  I see more and more posts about the usage of CNN in combination with LSTM, but I can't find any information about the advantages and disadvantages of using these in combination.
  
  This post ([Advantages of CNN vs. LSTM for sequence data like text or log-files](https://ai.stackexchange.com/questions/27312/advantages-of-cnn-vs-lstm-for-sequence-data-like-text-or-log-files)), it is asked about the advantages of CNN vs. LSTM. But I would like to know the advantages and disadvantages of adding CNN to LSTM for forecasting univariate sequential data? Or should you use one of the two algorithms?
  
  **Tags:** neural-networks, convolutional-neural-networks, long-short-term-memory
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-06-13. Score: 1 | Accepted answer
  
  If your data are 2D + time then you might want to use something like [ConvLSTM](https://arxiv.org/abs/1506.04214).
  
  If you only care about 1D + time then you don't need to add CNN to LSTM you only use one or the other. In terms of pros and cons have a look at [this empirical study](https://arxiv.org/abs/1803.01271) on how dilated convolutions compare to LSTMs for modeling sequential data. If you're also interested in the more theoretical aspects, [this paper](https://arxiv.org/abs/1810.06682) shows how temporal convolutional networks are related to truncated RNNs.
  
  
  ---
  
  ### Answer by username_2 on 2022-11-10. Score: 1
  
  A convolution layer can extract local patterns from its input (from the previous layer). Neighborhood size of the extracted pattern is relative to the kernel size. As we go deeper in the network the reception field of the layers increase. Accordingly, layers can recognize patters from larger neighborhoods of the input data. However, this increase in the reception field is very gradual. So, to understand patterns concerning large neighborhoods of input data, you need very deep networks. To ease this requirement, you can use dilated convolution layers, as in [1,2]. Hence, with shallower networks u can have reception fields as large as the whole input sequence. notice that many time series problems require patterns concerning the whole input sequence.   
  using LSTM layers in combination with convolution layers is also beneficial in many cases. in this scenario, several blocks of convolution layers first extract local hierarchical patterns. So the input sequence is represented by a reach collection of local patterns. Then, this patterns are feed to an LSTM to understand dependencies related to whole sequence. also, notice that an lstm layer generally requires more parameters to learn.   
  [1] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P. A. (2019). Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4), 917-963.  
  [2] Wang, Z., Yan, W., & Oates, T. (2017, May). Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) (pp. 1578-1585). IEEE.
  
  
  ---
  
  