<p>Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:</p>
<pre><code>self.embedding_layer = nn.Linear(4, d_model)
</code></pre>
<p>Where <code>d_model</code> is the dimension of the input to the transformer.</p>
<p><code>PositionalEncoding</code> is still needed so as to have a representation of time in the inputs.</p>
<pre><code>src = self.embedding_layer(src)
src = self.pos_encoding_layer(src)
output = self.transformer(src)
</code></pre>

CC BY-SA 4.0
2024-12-03T10:35:14.580Z
Is linear layer same as dense?
Muhammad Ikhwan Perwira
CC BY-SA 4.0
2021-07-14T16:59:03.173Z
2021-07-14T16:59:03.173Z
Dylan Kerler
<p>What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock <a href="https://keras.io/examples/nlp/text_classification_with_transformer/" rel="nofollow noreferrer">here</a>.</p>
<p>Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape-related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to the LSTM benchmark with the same data.</p>
<p>I am unsure if the positional encoding is still needed.</p>
<p>Overall, my question is, what is the proper way to process continuous sequence data, such as time-series, using the Transformer architecture?</p>

<deep-learning><keras><transformer><time-series><sequence-modeling>
What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
meliksahturker
# What is the proper way to process continuous sequence data, such as time-series, using the Transformer?

(Asked by: username_0 on 2021-07-11)

What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock [here](https://keras.io/examples/nlp/text_classification_with_transformer/).

Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape-related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to the LSTM benchmark with the same data.

I am unsure if the positional encoding is still needed.

Overall, my question is, what is the proper way to process continuous sequence data, such as time-series, using the Transformer architecture?

**Tags:** deep-learning, keras, transformer, time-series, sequence-modeling


---

## 1 answers

### Answer by username_1 on 2021-07-14. Score: 4

Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:
self.embedding_layer = nn.Linear(4, d_model)

Where d_model is the dimension of the input to the transformer.
PositionalEncoding is still needed so as to have a representation of time in the inputs.
src = self.embedding_layer(src)
src = self.pos_encoding_layer(src)
output = self.transformer(src)

> **Comments:**
> **username_2**: Is linear layer same as dense?  

---

