Answers.Body: <p>Taking the example you provided, you can use a simple RNN or any Seq-Seq architecture. Your input should be of the shape (BATCH_SIZE, NO_OF_TIMESTEPS, FEATURES_AT_EACH_TIMESTEP). So, you basically need to reformat your input as <code>arr = np.array([[1,4,7], [2,5,8], [3,6,9]])</code> - This assuming that 1,4,7 are representing different set of features at the same time-step (or same dim of diff vectors). Your training output (<strong>Y</strong>) would be of the shape (BATCH_SIZE, TIME_STEPS) OR <code>arr_y = np.array([[10, 3, 8]]) Then, you can have the model predict the output for the next 3 time-steps (or 3 dimensions of the vector). Note that, in your case you are not really constraining the model to look at the link [TF_simpleRNN][1]. Note that you need to use </code>return_sequences=True` and use a single unit as the output</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T04:40:47.833Z
Answers.Text: I did try to use [1,4,7], [2,5,8], [3,6,9] as feature. Basically, lets assume batch_size=1, then my input is (1, 3, 3). One round training is basically
  
  
  '''[3,6,9] -> [2,5,8] -> [1,4,7] -> [MODEL] -> [10,3,8]''' (first three arrows means they get popped into model one after another)
  
  
  I think I did implement in the right way, but it seems like, the model can not efficiently learn a certain rule, even as simple as vector XOR operation
Answers.UserDisplayName: Edee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T04:46:31.490Z
Answers.Text: some of the people say, flat(30, 30) into (1,900) will not impact RNN training, but to me, I think it does effect the RNN training, because flatting (30, 30) -> (1, 900) even change the timestamp and the meaning. I believe mathematically these two has no difference when training, but I just don't feel right.
Answers.UserDisplayName: Edee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T03:46:01.570Z
Answers.LastActivityDate: 2022-06-22T03:46:01.570Z
Answers.UserDisplayName: rahul bahadur
Body: <p>I am totally newbie into serial prediction.
  I am think about which model or AI paradigm can be used to do vector to vector prediction?</p>
  <p>For instance, [1,0,1] ^ [0,1,0] = [1,1,1]
  Another example could be: [1,0,1]^[0,1,0]^[1,1,0]^... = [RESULT]</p>
  <p>(I have to say that this example is not proper enough, because obviously we can use a multi-layer model to simply learn the rule of XOR operation with 100% accuracy, but I think it still can indicate what I want. The point is multi-dimensional input and figure out a non-linear rule. Let's say each row vector can be a row of a matrix, I want to find some pattern across each column entries.)</p>
  <p>Another better example:
  [[1,2,3], [4,5,6], [7,8,9]] =&gt; [10,3,8]</p>
  <p>I know seq2seq model, to some extend can do this job, so as pointer network. But I am just skeptical on my knowledge in this RNN area. And I do need some help.</p>
  <p>For the record, I think what I want is different from stock prediction or word prediction or sentence classification. My argue is that these tasks will need an embedding layer to output a one-hot or whatever vectorization and feed the embedding to the model.</p>
  <p>But my original input has already been vectors. I did try to feed those vectors as an embedding to the model. But theoretically, it does not make sense to me. And it did not work under my practice.</p>
  <p>Please enlighten me.</p>
  <p>Yes, I dare to share my code. Please be easy on me.</p>
  <pre><code>    # model structure
      model = keras.Sequential()
      model.add(Input(shape=(node_size, node_size)))  # seq, input_dim
      # The output of GRU will be a 3D tensor of shape (batch_size, timesteps, 256)
      model.add(LSTM(1024, return_sequences=True, activation=&quot;relu&quot;))
      # The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128)
      model.add(Dropout(0.5))
      model.add(LSTM(node_size, activation=&quot;relu&quot;))
      model.add(Dropout(0.5))
      # model.add(Dense(128, activation=&quot;relu&quot;))
      # model.add(Dropout(0.1))
      model.add(Dense(node_size, activation=&quot;sigmoid&quot;))
  
      # build model
      opt = keras.optimizers.Adam(learning_rate=0.001)
      model.compile(optimizer=opt, loss=&quot;binary_crossentropy&quot;, metrics=[BinaryAccuracy()])
      model.summary()
      # A_input = A_input.reshape(1, node_size, node_size)
      # label = label.reshape(1, node_size)
  
      # train
      model.fit(A_input, label, epochs=200, batch_size=10, validation_split=0.3)
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <recurrent-neural-networks><time-series><prediction>
Title: What model can solve vector to vector prediction?
UserDisplayName: Edee
ThreadText: # What model can solve vector to vector prediction?
  
  (Asked by: username_0 on 2022-06-22)
  
  I am totally newbie into serial prediction.
  I am think about which model or AI paradigm can be used to do vector to vector prediction?
  
  For instance, [1,0,1] ^ [0,1,0] = [1,1,1]
  Another example could be: [1,0,1]^[0,1,0]^[1,1,0]^... = [RESULT]
  
  (I have to say that this example is not proper enough, because obviously we can use a multi-layer model to simply learn the rule of XOR operation with 100% accuracy, but I think it still can indicate what I want. The point is multi-dimensional input and figure out a non-linear rule. Let's say each row vector can be a row of a matrix, I want to find some pattern across each column entries.)
  
  Another better example:
  [[1,2,3], [4,5,6], [7,8,9]] => [10,3,8]
  
  I know seq2seq model, to some extend can do this job, so as pointer network. But I am just skeptical on my knowledge in this RNN area. And I do need some help.
  
  For the record, I think what I want is different from stock prediction or word prediction or sentence classification. My argue is that these tasks will need an embedding layer to output a one-hot or whatever vectorization and feed the embedding to the model.
  
  But my original input has already been vectors. I did try to feed those vectors as an embedding to the model. But theoretically, it does not make sense to me. And it did not work under my practice.
  
  Please enlighten me.
  
  Yes, I dare to share my code. Please be easy on me.
  
  ```
      # model structure
      model = keras.Sequential()
      model.add(Input(shape=(node_size, node_size)))  # seq, input_dim
      # The output of GRU will be a 3D tensor of shape (batch_size, timesteps, 256)
      model.add(LSTM(1024, return_sequences=True, activation="relu"))
      # The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128)
      model.add(Dropout(0.5))
      model.add(LSTM(node_size, activation="relu"))
      model.add(Dropout(0.5))
      # model.add(Dense(128, activation="relu"))
      # model.add(Dropout(0.1))
      model.add(Dense(node_size, activation="sigmoid"))
  
      # build model
      opt = keras.optimizers.Adam(learning_rate=0.001)
      model.compile(optimizer=opt, loss="binary_crossentropy", metrics=[BinaryAccuracy()])
      model.summary()
      # A_input = A_input.reshape(1, node_size, node_size)
      # label = label.reshape(1, node_size)
  
      # train
      model.fit(A_input, label, epochs=200, batch_size=10, validation_split=0.3)
  ```
  
  ```
  
  **Tags:** recurrent-neural-networks, time-series, prediction
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-22. Score: 0
  
  Taking the example you provided, you can use a simple RNN or any Seq-Seq architecture. Your input should be of the shape (BATCH\_SIZE, NO\_OF\_TIMESTEPS, FEATURES\_AT\_EACH\_TIMESTEP). So, you basically need to reformat your input as `arr = np.array([[1,4,7], [2,5,8], [3,6,9]])` - This assuming that 1,4,7 are representing different set of features at the same time-step (or same dim of diff vectors). Your training output (**Y**) would be of the shape (BATCH\_SIZE, TIME\_STEPS) OR `arr_y = np.array([[10, 3, 8]]) Then, you can have the model predict the output for the next 3 time-steps (or 3 dimensions of the vector). Note that, in your case you are not really constraining the model to look at the link [TF_simpleRNN][1]. Note that you need to use` return\_sequences=True` and use a single unit as the output
  
  > **Comments:**
  > **username_0**: I did try to use [1,4,7], [2,5,8], [3,6,9] as feature. Basically, lets assume batch_size=1, then my input is (1, 3, 3). One round training is basically   '''[3,6,9] -> [2,5,8] -> [1,4,7] -> [MODEL] -> [10,3,8]''' (first three arrows means they get popped into model one after another)   I think I did implement in the right way, but it seems like, the model can not efficiently learn a certain rule, even as simple as vector XOR operation  
  > **username_0**: some of the people say, flat(30, 30) into (1,900) will not impact RNN training, but to me, I think it does effect the RNN training, because flatting (30, 30) -> (1, 900) even change the timestamp and the meaning. I believe mathematically these two has no difference when training, but I just don't feel right.  
  
  ---
  
  