Answers.Body: <p>In NLP, words are tokenized to unique integer tokens. These indices are subject to an embedding table, which basically &quot;converts&quot; a token into a (learnable) high-dimensional, continuous, dense vector.</p>
  <p>These dense vectors (or embeddings) are the representation learned by the transformer model to represent the different words. Now, if you want to use continuous input features you need to drop the embedding table and feed them directly to the transformer blocks.</p>
  <p>Moreover, the continuous input can be problematic depending on you task (e.g., classification, regression, or auto-regressive generation). In particular, in NPL the task is often to predict the next word (i.e., auto-regressive generation) which entails a classification head and learning objective over the discrete tokens (classes). Therefore, you may need to figure out a suitable output head for your model in order to accept continuous features.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-22T11:18:20.920Z
Answers.LastActivityDate: 2024-11-22T11:18:20.920Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>For timeseries that consists of amplitude, and time domain.</p>
  <p>For amplitude, the data is encoded into bins for every ±0.1 increment.</p>
  <p>E.g. Amplitude value range 0.1 until 0.2 is Token with index 1, 0.0 until 0.1 is token with index 0.</p>
  <p>So, it's similar like NLP</p>
  <p>But, what is the different if I directly feed with direct continous data into transformer?</p>
  
Tags: <transformer><data-preprocessing>
Title: What is the different between feeding transformer with continous data and with bins data (token) for timeseries?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # What is the different between feeding transformer with continous data and with bins data (token) for timeseries?
  
  (Asked by: username_0 on 2024-11-21)
  
  For timeseries that consists of amplitude, and time domain.
  
  For amplitude, the data is encoded into bins for every ±0.1 increment.
  
  E.g. Amplitude value range 0.1 until 0.2 is Token with index 1, 0.0 until 0.1 is token with index 0.
  
  So, it's similar like NLP
  
  But, what is the different if I directly feed with direct continous data into transformer?
  
  **Tags:** transformer, data-preprocessing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-22. Score: 2 | Accepted answer
  
  In NLP, words are tokenized to unique integer tokens. These indices are subject to an embedding table, which basically "converts" a token into a (learnable) high-dimensional, continuous, dense vector.
  
  These dense vectors (or embeddings) are the representation learned by the transformer model to represent the different words. Now, if you want to use continuous input features you need to drop the embedding table and feed them directly to the transformer blocks.
  
  Moreover, the continuous input can be problematic depending on you task (e.g., classification, regression, or auto-regressive generation). In particular, in NPL the task is often to predict the next word (i.e., auto-regressive generation) which entails a classification head and learning objective over the discrete tokens (classes). Therefore, you may need to figure out a suitable output head for your model in order to accept continuous features.
  
  
  ---
  
  