Answers.Body: <p>I think your input should have shape <code>[batch_size, time_points, n_features]</code> which would correspond to <code>[batch_size, sentance_length, embedding_dim]</code> in the standard transformer.</p>
  <p>The positional encoding would encode the ordering of the samples in time.</p>
  <p>The normal positional encoding I imagine is the standard thing to do. The only different thing to do would be to have a mpl transform the features into embedding vectors which then the positional encoding would be applied to.</p>
  <p>If you wanted to add the time as a feature I would embed it in the half unit circle. Normalize time to the interval <code>[0,1]</code> then do <code>torch.view_as_real(torch.exp(PI * normalized_time * 1j))</code> and append this to you features. The reason is that you want the dot product to be a similarity measure of the time embedding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T08:44:45.013Z
Answers.LastActivityDate: 2022-05-19T08:51:07.227Z
Answers.LastEditDate: 2022-05-19T08:51:07.227Z
Answers.UserDisplayName: Tom Huntington
Body: <p>I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension <code>[batch_size, n_features]</code>.</p>
  <p>I have a data point per day, so this is a time-series but there are no evidences of the time in my vectors and I know I must use positional encoding to tell my network “this vector is from the 21/05/2021, this one is from the 20/04/2019, etc”.</p>
  <p>The usual positional encoding with sin &amp; cos used in NLP doesn’t seem to fit my problem as it encodes the position relative to other words in the sentence and my features are independant values (the temperature of the day doesn’t come after the amount of rain for instance). Order of the data matters between two different inputs but not within the features.</p>
  <p>I don’t really know how to do encode my feature vectors for my transformer encoder to get something like <code>positional_encoded = original_vector + date_encoding</code> which I think could be great but I could also just add the day of year as a feature - would it be enough for a transformer?</p>
  <p>What would be the best way to do so?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-23T13:18:42.800Z
Comments.Text: Did you try absolute positional encoding?
Comments.UserDisplayName: emely_pi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-03T06:25:27.160Z
Comments.Text: same problem here, no news about the implementation? I'm trying to figure out how to solve it.
Comments.UserDisplayName: pabloverd
Tags: <transformer><time-series><positional-encoding>
Title: Positional Encoding of Time-Series features
UserDisplayName: Ouilliam
ThreadText: # Positional Encoding of Time-Series features
  
  (Asked by: username_0 on 2022-05-17)
  
  I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension `[batch_size, n_features]`.
  
  I have a data point per day, so this is a time-series but there are no evidences of the time in my vectors and I know I must use positional encoding to tell my network “this vector is from the 21/05/2021, this one is from the 20/04/2019, etc”.
  
  The usual positional encoding with sin & cos used in NLP doesn’t seem to fit my problem as it encodes the position relative to other words in the sentence and my features are independant values (the temperature of the day doesn’t come after the amount of rain for instance). Order of the data matters between two different inputs but not within the features.
  
  I don’t really know how to do encode my feature vectors for my transformer encoder to get something like `positional_encoded = original_vector + date_encoding` which I think could be great but I could also just add the day of year as a feature - would it be enough for a transformer?
  
  What would be the best way to do so?
  
  **Tags:** transformer, time-series, positional-encoding
  
  > **Comments:**
  > **username_1**: Did you try absolute positional encoding?  
  > **username_2**: same problem here, no news about the implementation? I'm trying to figure out how to solve it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-05-19. Score: 2
  
  I think your input should have shape `[batch_size, time_points, n_features]` which would correspond to `[batch_size, sentance_length, embedding_dim]` in the standard transformer.
  
  The positional encoding would encode the ordering of the samples in time.
  
  The normal positional encoding I imagine is the standard thing to do. The only different thing to do would be to have a mpl transform the features into embedding vectors which then the positional encoding would be applied to.
  
  If you wanted to add the time as a feature I would embed it in the half unit circle. Normalize time to the interval `[0,1]` then do `torch.view_as_real(torch.exp(PI * normalized_time * 1j))` and append this to you features. The reason is that you want the dot product to be a similarity measure of the time embedding.
  
  
  ---
  
  