Answers.Body: <p>It depends on the context bound limitation of your transformer and how temporal relations in your time series should be captured for your task.</p>
  <p>If your hardware and transformer can handle the memory and computational cost and if capturing long-range dependencies across the entire 5000 length time series is critical to your task such as detecting trends that span the entire series, then you could feed with <code>(1,5000)</code> shape.</p>
  <p>If your hardware or transformer cannot efficiently handle the sequence resulting from the quadratic time complexity of transformers and if local patterns within length 10 is sufficient for your task or you plan to aggregate results from all sliding windows to feed into another transformer or other time series forecasting model to capture global patterns, then you could use the second sliding window approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T19:09:48.037Z
Answers.LastActivityDate: 2024-11-29T19:09:48.037Z
Answers.UserDisplayName: cinch
Body: <p>Suppose I have time series with length of 5000.</p>
  <p>Transformer expects input with dimension of <code>(batch_size, sequence_length)</code> at the top of overview (excluding any hidden layer such as embedding dims, etc).</p>
  <p>Shall I just feed with <code>(1, 5000)</code>, or <a href="https://numpy.org/devdocs/reference/generated/numpy.lib.stride_tricks.sliding_window_view.html" rel="nofollow noreferrer">sliding window</a> for example with size <code>10</code>, therefore feeding it with shape <code>(4991, 10)</code>?</p>
  
Tags: <transformer><data-preprocessing><time-series>
Title: Is transformer still need windowed batch input?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Is transformer still need windowed batch input?
  
  (Asked by: username_0 on 2024-11-29)
  
  Suppose I have time series with length of 5000.
  
  Transformer expects input with dimension of `(batch_size, sequence_length)` at the top of overview (excluding any hidden layer such as embedding dims, etc).
  
  Shall I just feed with `(1, 5000)`, or [sliding window](https://numpy.org/devdocs/reference/generated/numpy.lib.stride_tricks.sliding_window_view.html) for example with size `10`, therefore feeding it with shape `(4991, 10)`?
  
  **Tags:** transformer, data-preprocessing, time-series
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-29. Score: 1 | Accepted answer
  
  It depends on the context bound limitation of your transformer and how temporal relations in your time series should be captured for your task.
  
  If your hardware and transformer can handle the memory and computational cost and if capturing long-range dependencies across the entire 5000 length time series is critical to your task such as detecting trends that span the entire series, then you could feed with `(1,5000)` shape.
  
  If your hardware or transformer cannot efficiently handle the sequence resulting from the quadratic time complexity of transformers and if local patterns within length 10 is sufficient for your task or you plan to aggregate results from all sliding windows to feed into another transformer or other time series forecasting model to capture global patterns, then you could use the second sliding window approach.
  
  
  ---
  
  