Answers.Body: <p>Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:</p>
  <pre><code>self.embedding_layer = nn.Linear(4, d_model)
  </code></pre>
  <p>Where <code>d_model</code> is the dimension of the input to the transformer.</p>
  <p><code>PositionalEncoding</code> is still needed so as to have a representation of time in the inputs.</p>
  <pre><code>src = self.embedding_layer(src)
  src = self.pos_encoding_layer(src)
  output = self.transformer(src)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T10:35:14.580Z
Answers.Text: Is linear layer same as dense?
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-14T16:59:03.173Z
Answers.LastActivityDate: 2021-07-14T16:59:03.173Z
Answers.UserDisplayName: Dylan Kerler
Body: <p>What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock <a href="https://keras.io/examples/nlp/text_classification_with_transformer/" rel="nofollow noreferrer">here</a>.</p>
  <p>Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape-related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to the LSTM benchmark with the same data.</p>
  <p>I am unsure if the positional encoding is still needed.</p>
  <p>Overall, my question is, what is the proper way to process continuous sequence data, such as time-series, using the Transformer architecture?</p>
  
Tags: <deep-learning><keras><transformer><time-series><sequence-modeling>
Title: What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
UserDisplayName: meliksahturker
ThreadText: # What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
  
  (Asked by: username_0 on 2021-07-11)
  
  What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock [here](https://keras.io/examples/nlp/text_classification_with_transformer/).
  
  Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape-related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to the LSTM benchmark with the same data.
  
  I am unsure if the positional encoding is still needed.
  
  Overall, my question is, what is the proper way to process continuous sequence data, such as time-series, using the Transformer architecture?
  
  **Tags:** deep-learning, keras, transformer, time-series, sequence-modeling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-14. Score: 4
  
  Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:
  
  ```
  self.embedding_layer = nn.Linear(4, d_model)
  
  ```
  
  Where `d_model` is the dimension of the input to the transformer.
  
  `PositionalEncoding` is still needed so as to have a representation of time in the inputs.
  
  ```
  src = self.embedding_layer(src)
  src = self.pos_encoding_layer(src)
  output = self.transformer(src)
  
  ```
  
  > **Comments:**
  > **username_2**: Is linear layer same as dense?  
  
  ---
  
  