Answers.Body: <p>Indeed transformer with <a href="https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Positional_encoding" rel="nofollow noreferrer">fixed-length positional encodings</a> such as your <code>SinePositionEncoding </code> doesn't inherently generalize well to sequence lengths that differ from those used during training, because they're periodic thus their values are tied to specific positions relative to a sequence. When you change the sequence length from 180 to 600, the model's learned dependencies on these encodings may not map correctly as the relationships between positions are <em>distorted</em>.</p>
  <p>To resolve this issue you may try <a href="https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Alternative_positional_encodings" rel="nofollow noreferrer">alternative positional embeddings</a> such as rotary positional embedding (RoPE) or relative position encodings used in Transformer-XL and T5.</p>
  <blockquote>
  <p>The benefit of RoPE is that the dot-product between two vectors depends on their relative location only</p>
  </blockquote>
  <blockquote>
  <p>Relative Position Encodings is similar to ALiBi, but more generic... This is contrasted with the original sinusoidal positional encoding, which is an &quot;absolute positional encoding&quot;.</p>
  </blockquote>
  <p>Or during inference, you could simply dynamically adjust the sine positional encodings to match the new sequence length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T23:46:02.320Z
Answers.Text: "...you could simply dynamically adjust the sine positional encodings to match the new sequence length." In keras3, I only see one parameter, which is **max_wavelength=10000**, https://keras.io/keras_hub/api/modeling_layers/sine_position_encoding/ So, is that something I need to adjusf?
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T00:12:17.927Z
Answers.Text: unfortunately not that easy, this is only a frequency-granularity reference parameter recommended in the original Transformer paper which is usually fine for most NLP tasks unless for very long input sequences you need to enlarge to prevent position encoding degradation or overlapping, not designed for dynamic input sequence length adjustment. You have to inject length argument via ```keras.Input((seq_len,), dtype="float32")```. Hope this clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T00:19:21.343Z
Answers.Text: I will try that `RoPE` and `AliBi`.
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T00:24:17.480Z
Answers.Text: Well, actually I can just keep use SinePE for my case, since it's trained by sliding window with `stride=1`. Then, during inference to reduce computation, just perform preprocessing reshape with padded first that divisible by `180`. `(1,600)->(1,720)->(4,180)` And then perform post-processing truncating the padding. `(4,180)->(1,720)->(1,600)`. With this, the model is perceving that as sliding window (though it's not sliding window, just reshaping tensor).
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T00:41:52.193Z
Answers.Text: Training LLM is an art so definitely you should try out different alternative ways to build up your experience with each method since no one fits all cases. Your chunk-split with padding method during inference time is also a popular method keeping the same SinePE for each chunk, though the added padding would affect efficiency and processing logic complication, and if your model depends on long range relations across split chunks, this might  introduce challenges as well as combining the generations with possible overlap.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-03T06:05:12.237Z
Answers.Text: I set `180` a.k.a 3 secs from domain specific knowledge of medical ECG signal, where the average heart beat of human is 1.2 seconds, and therefore I set the context window to the 3 secs a.k.a `180` points in 60Hz sample rate. So I think the model can still capture the long range relations, but not across split chunks, instead between heart beat.
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T23:02:31.657Z
Answers.LastActivityDate: 2024-12-02T23:02:31.657Z
Answers.UserDisplayName: cinch
Body: <p>Given (this <code>600</code> will be used for inference):</p>
  <ul>
  <li><code>X_infer</code> tensor with the shape of <code>(num_window, window_len) -&gt; (1, 600)</code></li>
  <li><code>y_infer</code> tensor with the shape of <code>(num_window, window_len) -&gt; (1, 600)</code></li>
  </ul>
  <p>Vocab size:</p>
  <ul>
  <li><code>X</code>: 128 (Amplitude)</li>
  <li><code>y</code>: 5 (Color Label)</li>
  </ul>
  <p>Performing <code>sliding_window_view</code> from origin, returns (this <code>180</code> will be used during training):</p>
  <ul>
  <li><code>X_train</code> tensor with the shape of <code>(num_window, window_len) -&gt; (471, 180)</code></li>
  <li><code>y_train</code> tensor with the shape of <code>(num_window, window_len) -&gt; (471, 180)</code></li>
  </ul>
  <p>Train:</p>
  <pre><code>model = instantiate_untrained_model(seq_len=180)
  model.fit(X_train, y_train)
  </code></pre>
  <p>Save (weights only):</p>
  <pre><code>model.save_weights('trained.weights.h5')
  </code></pre>
  <p>Infer with difference <code>seq_len</code>:</p>
  <pre><code># 600
  model = instantiate_untrained_model(seq_len=600) # entire array
  model.load_weights('trained.weights.h5') # No error
  y_pred_600 = model.predict(X_infer)
  
  # 180
  model = instantiate_untrained_model(seq_len=180) # windowed array
  model.load_weights('trained.weights.h5')
  y_pred_180 = model.predict(X_train)
  </code></pre>
  <p>Plotting Prediction <code>y_pred_180</code> and its ground truth:</p>
  <p><a href="https://i.sstatic.net/0kHa9JlC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0kHa9JlC.png" alt="enter image description here" /></a></p>
  <p>Plotting Prediction <code>y_pred_600</code> and its ground truth:</p>
  <p><a href="https://i.sstatic.net/jtE4fEvF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jtE4fEvF.png" alt="enter image description here" /></a></p>
  <p>Even though signal, but you can think this problem as Named Entity Recognition.</p>
  <p>And here is the keras model:</p>
  <pre><code>  SEQ_LEN=seq_len
    VOCAB_SIZE=128
    EMBEDD_DIM=128
  
    encoder_inputs = Input(shape=(SEQ_LEN,), name=&quot;encoder_inputs&quot;, dtype=np.uint8)
    token_embeddings = Embedding(input_dim=VOCAB_SIZE, output_dim=EMBEDD_DIM)(encoder_inputs)
    position_encodings = SinePositionEncoding()(token_embeddings)
  
    # this line adds up the embeddings and fixes the problem
    embeddings = token_embeddings + position_encodings
  
    encoder_outputs = TransformerEncoder(intermediate_dim=EMBEDD_DIM*4, num_heads=2, dropout=0.05)(inputs=embeddings)
  
    # Output layer for vocabulary size of 5
    output_predictions = Dense(units=5, activation=None)(encoder_outputs)
  
    # Final model
    model = Model(encoder_inputs, output_predictions, name=&quot;transformer_encoder&quot;)
  </code></pre>
  <p><strong>I expect during inference, the transformer model can accept arbitrary sequence length with good performance?</strong></p>
  <p>Why this happened? Is there relation with <code>SinePositionEncoding</code> or something? The model is not flexible and not robust, it's only good performance when the sequence length is <code>180</code> instead of arbitrary length? How can I solve this?</p>
  
Tags: <deep-learning><natural-language-processing><transformer><keras><performance>
Title: Transformer Encoder give worse performance when the sequence length during training is difference compared to inference
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Transformer Encoder give worse performance when the sequence length during training is difference compared to inference
  
  (Asked by: username_0 on 2024-12-01)
  
  Given (this 600 will be used for inference):
  
  X_infer tensor with the shape of (num_window, window_len) -> (1, 600)
  y_infer tensor with the shape of (num_window, window_len) -> (1, 600)
  
  Vocab size:
  
  X: 128 (Amplitude)
  y: 5 (Color Label)
  
  Performing sliding_window_view from origin, returns (this 180 will be used during training):
  
  X_train tensor with the shape of (num_window, window_len) -> (471, 180)
  y_train tensor with the shape of (num_window, window_len) -> (471, 180)
  
  Train:
  model = instantiate_untrained_model(seq_len=180)
  model.fit(X_train, y_train)
  
  Save (weights only):
  model.save_weights('trained.weights.h5')
  
  Infer with difference seq_len:
  # 600
  model = instantiate_untrained_model(seq_len=600) # entire array
  model.load_weights('trained.weights.h5') # No error
  y_pred_600 = model.predict(X_infer)
  
  # 180
  model = instantiate_untrained_model(seq_len=180) # windowed array
  model.load_weights('trained.weights.h5')
  y_pred_180 = model.predict(X_train)
  
  Plotting Prediction y_pred_180 and its ground truth:
  
  Plotting Prediction y_pred_600 and its ground truth:
  
  Even though signal, but you can think this problem as Named Entity Recognition.
  And here is the keras model:
    SEQ_LEN=seq_len
    VOCAB_SIZE=128
    EMBEDD_DIM=128
  
    encoder_inputs = Input(shape=(SEQ_LEN,), name="encoder_inputs", dtype=np.uint8)
    token_embeddings = Embedding(input_dim=VOCAB_SIZE, output_dim=EMBEDD_DIM)(encoder_inputs)
    position_encodings = SinePositionEncoding()(token_embeddings)
  
    # this line adds up the embeddings and fixes the problem
    embeddings = token_embeddings + position_encodings
  
    encoder_outputs = TransformerEncoder(intermediate_dim=EMBEDD_DIM*4, num_heads=2, dropout=0.05)(inputs=embeddings)
  
    # Output layer for vocabulary size of 5
    output_predictions = Dense(units=5, activation=None)(encoder_outputs)
  
    # Final model
    model = Model(encoder_inputs, output_predictions, name="transformer_encoder")
  
  I expect during inference, the transformer model can accept arbitrary sequence length with good performance?
  Why this happened? Is there relation with SinePositionEncoding or something? The model is not flexible and not robust, it's only good performance when the sequence length is 180 instead of arbitrary length? How can I solve this?
  
  **Tags:** deep-learning, natural-language-processing, transformer, keras, performance
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-02. Score: 1 | Accepted answer
  
  Indeed transformer with [fixed-length positional encodings](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Positional_encoding) such as your `SinePositionEncoding`  doesn't inherently generalize well to sequence lengths that differ from those used during training, because they're periodic thus their values are tied to specific positions relative to a sequence. When you change the sequence length from 180 to 600, the model's learned dependencies on these encodings may not map correctly as the relationships between positions are *distorted*.
  
  To resolve this issue you may try [alternative positional embeddings](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Alternative_positional_encodings) such as rotary positional embedding (RoPE) or relative position encodings used in Transformer-XL and T5.
  
  > The benefit of RoPE is that the dot-product between two vectors depends on their relative location only
  
  > Relative Position Encodings is similar to ALiBi, but more generic... This is contrasted with the original sinusoidal positional encoding, which is an "absolute positional encoding".
  
  Or during inference, you could simply dynamically adjust the sine positional encodings to match the new sequence length.
  
  > **Comments:**
  > **username_0**: "...you could simply dynamically adjust the sine positional encodings to match the new sequence length." In keras3, I only see one parameter, which is **max_wavelength=10000**, https://keras.io/keras_hub/api/modeling_layers/sine_position_encoding/ So, is that something I need to adjusf?  
  > **username_1**: unfortunately not that easy, this is only a frequency-granularity reference parameter recommended in the original Transformer paper which is usually fine for most NLP tasks unless for very long input sequences you need to enlarge to prevent position encoding degradation or overlapping, not designed for dynamic input sequence length adjustment. You have to inject length argument via ```keras.Input((seq_len,), dtype="float32")```. Hope this clarifies.  
  > **username_0**: I will try that `RoPE` and `AliBi`.  
  > **username_0**: Well, actually I can just keep use SinePE for my case, since it's trained by sliding window with `stride=1`. Then, during inference to reduce computation, just perform preprocessing reshape with padded first that divisible by `180`. `(1,600)->(1,720)->(4,180)` And then perform post-processing truncating the padding. `(4,180)->(1,720)->(1,600)`. With this, the model is perceving that as sliding window (though it's not sliding window, just reshaping tensor).  
  > **username_1**: Training LLM is an art so definitely you should try out different alternative ways to build up your experience with each method since no one fits all cases. Your chunk-split with padding method during inference time is also a popular method keeping the same SinePE for each chunk, though the added padding would affect efficiency and processing logic complication, and if your model depends on long range relations across split chunks, this might  introduce challenges as well as combining the generations with possible overlap.  
  > **username_0**: I set `180` a.k.a 3 secs from domain specific knowledge of medical ECG signal, where the average heart beat of human is 1.2 seconds, and therefore I set the context window to the 3 secs a.k.a `180` points in 60Hz sample rate. So I think the model can still capture the long range relations, but not across split chunks, instead between heart beat.  
  
  ---
  
  