Answers.Body: <p>If each of your three features is a scalar then my first attempt would be to combine them into a vector for each step in the sequence. So instead of <code>LSTM(128, input_shape=(30,1))</code> for a length-30 univariate sequence you would say <code>LSTM(128, input_shape=(30,3))</code> for a multivariate (3) sequence. Similarly your output would become <code>TimeDistributed(Dense(3, activation='linear'))</code>.</p>
  <p>If your input features are each a vector then you should consider switching from the sequential API to the functional API. You would need three separate Input objects followed by a Concatenate layer before the LSTM layer. You would have three Dense layers to be your corresponding outputs. I tried adapting your code:</p>
  <pre><code>in0 = Input(shape=(trainX0.shape[1], trainX0.shape[2]))
  in1 = Input(shape=(trainX1.shape[1], trainX1.shape[2]))
  in2 = Input(shape=(trainX2.shape[1], trainX2.shape[2]))
  concat = Concatenate()([in0, in1, in2])
  lstm_enc = LSTM(128)(concat)
  lstm_enc_drop = Dropout(rate=0.2)(lstm_enc)
  rep = RepeatVector(trainX0.shape[1])(lstm_enc_drop)
  lstm_dec = LSTM(128, return_sequences=True)(rep)
  lstm_dec_drop = Dropout(rate=0.2)(lstm_dec)
  out0 = TimeDistributed(Dense(trainX0.shape[2], activation='linear'))(lstm_dec_drop)
  out1 = TimeDistributed(Dense(trainX1.shape[2], activation='linear'))(lstm_dec_drop)
  out2 = TimeDistributed(Dense(trainX2.shape[2], activation='linear'))(lstm_dec_drop)
  model = Model(inputs=[in0, in1, in2], outputs=[out0, out1, out2])
  model.compile(optimizer='adam', loss='mae')
  model.summary()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T13:51:42.967Z
Answers.Text: My 3 inputs would be 3 columns of time series values e.g. 0.31314 0.15891 0.41713. Would the output of concatenate be a single vector i.e. [0.31314 0.15891 0.41713] or a single value?
Answers.UserDisplayName: Angelina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T18:27:31.013Z
Answers.Text: Concatenate would produce a 3-dimension vector. The output tensor from the Concatenate layer will be of form (batch, steps, features) so your model summary should report an output tensor shape of (None, 30, 3) given your 30 time steps and 3 features.
  
  I think that using multiple Inputs with Concatenate would be functionally equivalent to changing your input shape to (30, 3) and providing it with the 30x3 input data array. That way you wouldn't need to worry about changing from Sequential to Functional.
Answers.UserDisplayName: CorruptedHeapScapeGoat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T13:16:23.397Z
Answers.LastActivityDate: 2021-03-18T13:16:23.397Z
Answers.UserDisplayName: CorruptedHeapScapeGoat
Body: <p>I have the following code snippet which takes in a single column of value i.e. 1 feature. How do I modify the LSTM model such that it accepts 3 features?</p>
  <pre><code>import numpy as np
  from keras.models import Sequential
  from keras.layers import LSTM, Input, Dropout
  from keras.layers import Dense
  from keras.layers import RepeatVector
  from keras.layers import TimeDistributed
  import pandas as pd
  from matplotlib import pyplot as plt
  from sklearn.preprocessing import MinMaxScaler, StandardScaler
  from keras.models import Model
  import seaborn as sns    
  
  dataframe = pd.read_csv('GE.csv')
  dataframe.head()
  
  df = dataframe[['Date', 'EnergyInWatts']]
  df['Date'] = pd.to_datetime(df['Date'])
  sns.lineplot(x=df['Date'], y=df['EnergyInWatts'])
  
  #train, test = df.loc[df['Date'] &lt;= '2003-12-31'], df.loc[df['Date'] &gt; '2003-12-31']
  train = df.loc[df['Date'] &lt;= '2003-12-31']
  test = df.loc[df['Date'] &gt; '2003-12-31']
  
  scaler = StandardScaler()
  
  scaler = scaler.fit(train[['EnergyInWatts']])
  
  train['EnergyInWatts'] = scaler.transform(train[['EnergyInWatts']])
  test['EnergyInWatts'] = scaler.transform(test[['EnergyInWatts']])
  
  seq_size = 30 
  
  
  def to_sequences(x, y, seq_size=1):
      x_values = []
      y_values = []
  
      for i in range(len(x)-seq_size):
          #print(i)
          x_values.append(x.iloc[i:(i+seq_size)].values)
          y_values.append(y.iloc[i+seq_size])
          
      return np.array(x_values), np.array(y_values)
  
  trainX, trainY = to_sequences(train[['EnergyInWatts']], train['EnergyInWatts'], seq_size)
  testX, testY = to_sequences(test[['EnergyInWatts']], test['EnergyInWatts'], seq_size)
  
  
  model = Sequential()
  model.add(LSTM(128, input_shape=(trainX.shape[1], trainX.shape[2])))
  model.add(Dropout(rate=0.2))
  model.add(RepeatVector(trainX.shape[1]))
  model.add(LSTM(128, return_sequences=True))
  model.add(Dropout(rate=0.2))
  model.add(TimeDistributed(Dense(trainX.shape[2])))
  model.compile(optimizer='adam', loss='mae')
  model.summary()
  
  history = model.fit(trainX, trainY, epochs=10, batch_size=32, validation_split=0.1, verbose=1)
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <machine-learning><long-short-term-memory><autoencoders>
Title: Convert LSTM univariate Autoencoder to multivariate Autoencoder
UserDisplayName: Angelina
ThreadText: # Convert LSTM univariate Autoencoder to multivariate Autoencoder
  
  (Asked by: username_0 on 2021-03-17)
  
  I have the following code snippet which takes in a single column of value i.e. 1 feature. How do I modify the LSTM model such that it accepts 3 features?
  import numpy as np
  from keras.models import Sequential
  from keras.layers import LSTM, Input, Dropout
  from keras.layers import Dense
  from keras.layers import RepeatVector
  from keras.layers import TimeDistributed
  import pandas as pd
  from matplotlib import pyplot as plt
  from sklearn.preprocessing import MinMaxScaler, StandardScaler
  from keras.models import Model
  import seaborn as sns    
  
  dataframe = pd.read_csv('GE.csv')
  dataframe.head()
  
  df = dataframe[['Date', 'EnergyInWatts']]
  df['Date'] = pd.to_datetime(df['Date'])
  sns.lineplot(x=df['Date'], y=df['EnergyInWatts'])
  
  #train, test = df.loc[df['Date'] <= '2003-12-31'], df.loc[df['Date'] > '2003-12-31']
  train = df.loc[df['Date'] <= '2003-12-31']
  test = df.loc[df['Date'] > '2003-12-31']
  
  scaler = StandardScaler()
  
  scaler = scaler.fit(train[['EnergyInWatts']])
  
  train['EnergyInWatts'] = scaler.transform(train[['EnergyInWatts']])
  test['EnergyInWatts'] = scaler.transform(test[['EnergyInWatts']])
  
  seq_size = 30 
  
  
  def to_sequences(x, y, seq_size=1):
      x_values = []
      y_values = []
  
      for i in range(len(x)-seq_size):
          #print(i)
          x_values.append(x.iloc[i:(i+seq_size)].values)
          y_values.append(y.iloc[i+seq_size])
          
      return np.array(x_values), np.array(y_values)
  
  trainX, trainY = to_sequences(train[['EnergyInWatts']], train['EnergyInWatts'], seq_size)
  testX, testY = to_sequences(test[['EnergyInWatts']], test['EnergyInWatts'], seq_size)
  
  
  model = Sequential()
  model.add(LSTM(128, input_shape=(trainX.shape[1], trainX.shape[2])))
  model.add(Dropout(rate=0.2))
  model.add(RepeatVector(trainX.shape[1]))
  model.add(LSTM(128, return_sequences=True))
  model.add(Dropout(rate=0.2))
  model.add(TimeDistributed(Dense(trainX.shape[2])))
  model.compile(optimizer='adam', loss='mae')
  model.summary()
  
  history = model.fit(trainX, trainY, epochs=10, batch_size=32, validation_split=0.1, verbose=1)
  ```
  
  **Tags:** machine-learning, long-short-term-memory, autoencoders
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-18. Score: 0 | Accepted answer
  
  If each of your three features is a scalar then my first attempt would be to combine them into a vector for each step in the sequence. So instead of LSTM(128, input_shape=(30,1)) for a length-30 univariate sequence you would say LSTM(128, input_shape=(30,3)) for a multivariate (3) sequence. Similarly your output would become TimeDistributed(Dense(3, activation='linear')).
  If your input features are each a vector then you should consider switching from the sequential API to the functional API. You would need three separate Input objects followed by a Concatenate layer before the LSTM layer. You would have three Dense layers to be your corresponding outputs. I tried adapting your code:
  in0 = Input(shape=(trainX0.shape[1], trainX0.shape[2]))
  in1 = Input(shape=(trainX1.shape[1], trainX1.shape[2]))
  in2 = Input(shape=(trainX2.shape[1], trainX2.shape[2]))
  concat = Concatenate()([in0, in1, in2])
  lstm_enc = LSTM(128)(concat)
  lstm_enc_drop = Dropout(rate=0.2)(lstm_enc)
  rep = RepeatVector(trainX0.shape[1])(lstm_enc_drop)
  lstm_dec = LSTM(128, return_sequences=True)(rep)
  lstm_dec_drop = Dropout(rate=0.2)(lstm_dec)
  out0 = TimeDistributed(Dense(trainX0.shape[2], activation='linear'))(lstm_dec_drop)
  out1 = TimeDistributed(Dense(trainX1.shape[2], activation='linear'))(lstm_dec_drop)
  out2 = TimeDistributed(Dense(trainX2.shape[2], activation='linear'))(lstm_dec_drop)
  model = Model(inputs=[in0, in1, in2], outputs=[out0, out1, out2])
  model.compile(optimizer='adam', loss='mae')
  model.summary()
  
  > **Comments:**
  > **username_0**: My 3 inputs would be 3 columns of time series values e.g. 0.31314 0.15891 0.41713. Would the output of concatenate be a single vector i.e. [0.31314 0.15891 0.41713] or a single value?  
  > **username_1**: Concatenate would produce a 3-dimension vector. The output tensor from the Concatenate layer will be of form (batch, steps, features) so your model summary should report an output tensor shape of (None, 30, 3) given your 30 time steps and 3 features.  I think that using multiple Inputs with Concatenate would be functionally equivalent to changing your input shape to (30, 3) and providing it with the 30x3 input data array. That way you wouldn't need to worry about changing from Sequential to Functional.  
  
  ---
  
  