Body: <p>I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.</p>
  <p>As I just experiment with the positional encoding portion of the code I set up a toy model: I generated a time series that contains the log changes of a sine function and run a classification model that predicts whether the subsequent value is positive or negative. Simple enough. I also added a few time series with random walks to try to throw off the model.</p>
  <p>Predictably, the model very quickly reaches a categorical accuracy of around 99%. Without positional encoding that happens already in the 3rd epoch. However, with positional encoding (I use the same implementation as proposed in the &quot;Attention is all you need&quot; paper), it takes over 100 epochs to reach a similar accuracy level.</p>
  <p>So, clearly, all else being equal, learning with positional encoding takes much longer to reach an equal accuracy level than without positional encoding.</p>
  <p>Has anyone witnessed similar observations? Apparently adding the positional encodings to the actual values seems to confuse the model. I have not tried concatenations yet. Any advice?</p>
  <p>Edit: Or does it simply mean that learned positional encodings perform better than sin/cos encodings? I have not made any special provisions to encourage learned positional encodings, I simply either added the positional encodings to the actual values or I did not.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T08:34:51.303Z
Comments.Text: I wonder what kind of positional encoding you used. If you just add, like I think the Transformer paper did, then it makes perfect sense that it's harder to train because that basically scrambles all the word vectors based on position. Perhaps concatenating could work better, but you'd have more weights to learn.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T23:52:18.583Z
Comments.Text: @user253751, which then begs the question of the need for postional encodings. I definitely reach higher accuracy levels and that a lot earlier without positional encodings as applied to multivariate time series
Comments.UserDisplayName: Matt
Tags: <tensorflow><pytorch><transformer><time-series><positional-encoding>
Title: Positional Encoding in Transformer on multi-variate time series data hurts performance
UserDisplayName: Matt
ThreadText: # Positional Encoding in Transformer on multi-variate time series data hurts performance
  
  (Asked by: username_0 on 2021-07-27)
  
  I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.
  
  As I just experiment with the positional encoding portion of the code I set up a toy model: I generated a time series that contains the log changes of a sine function and run a classification model that predicts whether the subsequent value is positive or negative. Simple enough. I also added a few time series with random walks to try to throw off the model.
  
  Predictably, the model very quickly reaches a categorical accuracy of around 99%. Without positional encoding that happens already in the 3rd epoch. However, with positional encoding (I use the same implementation as proposed in the "Attention is all you need" paper), it takes over 100 epochs to reach a similar accuracy level.
  
  So, clearly, all else being equal, learning with positional encoding takes much longer to reach an equal accuracy level than without positional encoding.
  
  Has anyone witnessed similar observations? Apparently adding the positional encodings to the actual values seems to confuse the model. I have not tried concatenations yet. Any advice?
  
  Edit: Or does it simply mean that learned positional encodings perform better than sin/cos encodings? I have not made any special provisions to encourage learned positional encodings, I simply either added the positional encodings to the actual values or I did not.
  
  **Tags:** tensorflow, pytorch, transformer, time-series, positional-encoding
  
  > **Comments:**
  > **username_1**: I wonder what kind of positional encoding you used. If you just add, like I think the Transformer paper did, then it makes perfect sense that it's harder to train because that basically scrambles all the word vectors based on position. Perhaps concatenating could work better, but you'd have more weights to learn.  
  > **username_0**: @user253751, which then begs the question of the need for postional encodings. I definitely reach higher accuracy levels and that a lot earlier without positional encodings as applied to multivariate time series  
  
  ---
  
  ## 0 answers
  
  