Answers.Body: <p>If you know for a fact that you will always have 64 points in the time series, then there is no advantage of using a transformer over a MLP.</p>
  <p>The advantage of the transformer is it can batch process variable length sequences. For example if the input dimension was <code>(n,4)</code> where <code>n</code> is variable, you could use a transformer encoder, pool the inputs to a fixed length embedding, then send the embedding to a classification MLP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-10T04:50:44.150Z
Answers.Text: Thank you for the answer. As far as I have learnt, the transformer encoder always gives the output of same dimension as the input passed to it. How do I pool variable length inputs and get a fixed length output?
Answers.UserDisplayName: Zohaib Hamdule
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-10T00:32:57.717Z
Answers.LastActivityDate: 2023-10-10T00:32:57.717Z
Answers.UserDisplayName: Karl
Body: <p>Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For that I am first passing these tensors into a Transformer Encoder (having 2 attention heads and 2 encoder layers) that outputs a tensor of the same dimension. This output tensor is being flattened and passes onto a dense layer for classification. Is there some advantage of passing the time series through the encoder and classifying the encoded output over directly passing the original tensors to the dense layer.</p>
  <p>I tried this experimentally and saw no significant increase in accuracy when using the transformer encoder. However, the data I had was quite simple and not enough to make any conclusions. Also an expert I know insists that the model with the input processed by a transformer should work better.</p>
  <p>One thing I observed was a steeper decrease in loss when using the encoded tensors for classification.</p>
  <p>I also referred to this resource on this matter: <a href="https://www.linkedin.com/pulse/time-series-classification-model-based-transformer-gokmen/" rel="nofollow noreferrer">https://www.linkedin.com/pulse/time-series-classification-model-based-transformer-gokmen/</a></p>
  
Tags: <neural-networks><machine-learning><transformer><dense-layers>
Title: Time Series Classification using Transformer Encoder
UserDisplayName: Zohaib Hamdule
ThreadText: # Time Series Classification using Transformer Encoder
  
  (Asked by: username_0 on 2023-10-09)
  
  Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For that I am first passing these tensors into a Transformer Encoder (having 2 attention heads and 2 encoder layers) that outputs a tensor of the same dimension. This output tensor is being flattened and passes onto a dense layer for classification. Is there some advantage of passing the time series through the encoder and classifying the encoded output over directly passing the original tensors to the dense layer.
  
  I tried this experimentally and saw no significant increase in accuracy when using the transformer encoder. However, the data I had was quite simple and not enough to make any conclusions. Also an expert I know insists that the model with the input processed by a transformer should work better.
  
  One thing I observed was a steeper decrease in loss when using the encoded tensors for classification.
  
  I also referred to this resource on this matter: <https://www.linkedin.com/pulse/time-series-classification-model-based-transformer-gokmen/>
  
  **Tags:** neural-networks, machine-learning, transformer, dense-layers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-10. Score: 2
  
  If you know for a fact that you will always have 64 points in the time series, then there is no advantage of using a transformer over a MLP.
  
  The advantage of the transformer is it can batch process variable length sequences. For example if the input dimension was `(n,4)` where `n` is variable, you could use a transformer encoder, pool the inputs to a fixed length embedding, then send the embedding to a classification MLP.
  
  > **Comments:**
  > **username_0**: Thank you for the answer. As far as I have learnt, the transformer encoder always gives the output of same dimension as the input passed to it. How do I pool variable length inputs and get a fixed length output?  
  
  ---
  
  