Answers.Body: <p>I got this batch-based GRU to work:</p>
  <pre><code>import numpy as np
  import tensorflow as tf
  
  units = 2
  seq_length = 32
  
  X = (np.random.rand(int(1e4), seq_length, 1) &gt; 0.5).astype(float)
  Y = 1 - X
  
  model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=X.shape[1:]),
    tf.keras.layers.GRU(units, return_sequences=True),
    tf.keras.layers.Conv1D(1, 1, activation='linear')
  ])
  
  print(model.summary())
  
  model.compile(optimizer='adam', loss='mean_squared_error')
  history = model.fit(X, Y, batch_size=512, epochs=10, verbose=0, validation_split=0.1)
  
  for k in sorted(history.history.keys()):
      print('%15s: %s' % (k, str(np.array(history.history[k][:10]).round(3))))
  print()
  
  history = model.fit(X, Y, batch_size=512, epochs=40, verbose=0, validation_split=0.1)
  
  for k in sorted(history.history.keys()):
      print('%15s: %s' % (k, str(np.array(history.history[k][-10:]).round(3))))
  </code></pre>
  <p>It trains for 10 epochs, prints logs, trains for 40 more epochs and prints logs of the last 10 epochs.</p>
  <p>Here is an example run:</p>
  <pre><code>Model: &quot;sequential_104&quot;
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  gru_39 (GRU)                 (None, 32, 2)             30        
  _________________________________________________________________
  conv1d_68 (Conv1D)           (None, 32, 1)             3         
  =================================================================
  Total params: 33
  Trainable params: 33
  Non-trainable params: 0
  _________________________________________________________________
  
             loss: [1.137 0.97  0.832 0.718 0.625 0.546 0.479 0.422 0.372 0.329]
         val_loss: [1.041 0.891 0.767 0.665 0.58  0.508 0.447 0.394 0.348 0.307]
  
             loss: [0.026 0.023 0.021 0.019 0.017 0.016 0.014 0.013 0.011 0.01 ]
         val_loss: [0.024 0.022 0.02  0.018 0.016 0.015 0.013 0.012 0.011 0.009]
  </code></pre>
  <p>Using 8 units helps it to converge faster, after 10 epochs the loss is about 0.1.</p>
  <p>I noticed that your network outputs the GRU's state directly as the predicted value.</p>
  <p><em>Maybe there is a problem since the output dimension doesn't match the target value? There might be some broadcasting going on, which hides the issue.</em> <strong>&lt;-</strong> edit: my earlier assumption, it isn't correct.</p>
  <p>My network's <code>Conv1D</code> converts the GRU's <code>units</code> dimensional output to a <code>N x seq_length x 1</code> prediction. Adding it to your online version seems to fix it. Although the learning is quite unreliable, maybe due to the batch size of one.</p>
  <p><strong>Edit</strong>: Aah after reading your code more carefully I noticed that the target predicted value isn't the previous value of the input (a single digit) but this:</p>
  <pre><code>y_train = np.zeros(units) if prevIsBlack else np.ones(units)
  </code></pre>
  <p>I don't know a neural network architecture in which we'd measure the loss directly on the RNN's output, it is only used by network to &quot;remember&quot; data or state from the past. But it is then processed further by subsequent layers. First of all the GRU's activation funtion is <code>tanh</code>, so it cannot even produce values outside of the range from -1 to 1.</p>
  <p>I hope this clears things up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T14:01:09.693Z
Answers.LastActivityDate: 2022-01-06T14:12:49.867Z
Answers.LastEditDate: 2022-01-06T14:12:49.867Z
Answers.UserDisplayName: NikoNyrh
Body: <p>Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs.</p>
  <p>Consider a task where you need to predict <strong>0</strong> if the <strong>previous</strong> input was <strong>1</strong>. For example: <code>x=[0,1,0,0]</code>, <code>y=[1,1,0,1]</code>. We train the model <strong>online</strong>, i.e. the number of timesteps the RNN sees at a time is equal to one (in Keras you can make an LSTM/GRU cell stateful so that it passes state (<em>memory about previous inputs?</em>) between batches). And the problem is that, in my tests, RNN cells can not cope with this simple task.</p>
  <p>I suspect that my intuition about RNN is built wrong. I would appreciate someone could help me with that.</p>
  <p>Test script:</p>
  <pre><code>import numpy as np
  import tensorflow as tf
  from random import randrange
  
  batch_size = 1
  timesteps = 1
  size = 1
  units = 16
  shape = (timesteps, size)
  
  model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape = shape, batch_size = batch_size),
    tf.keras.layers.GRU(
      units,
      stateful = True,
      return_sequences = True
    )
  ])
  
  model.compile(optimizer='adam',
                loss='mean_squared_error')
  
  print(model.summary())
  
  SAME = False
  
  i = 0
  prevIsBlack = False
  x_batch, y_batch = [], []
  x_steps, y_steps = [], []
  
  while True:
    i += 1
  
    isBlack = randrange(4) == 2
  
    if SAME: prevIsBlack = isBlack 
  
    x_train = np.ones(size)   if isBlack else np.zeros(size)
    y_train = np.zeros(units) if prevIsBlack else np.ones(units)
  
    if not SAME: prevIsBlack = isBlack 
  
    x_steps.append(x_train)
    y_steps.append(y_train)
  
    if len(x_steps) &lt; timesteps:
      continue
  
    x_batch.append(np.stack(x_steps))
    y_batch.append(np.stack(y_steps))
  
    x_steps, y_steps = [], []
  
    if len(x_batch) &lt; batch_size:
      continue
  
    res = model.fit(np.stack(x_batch), np.stack(y_batch), 
      epochs = 1, 
      shuffle = False,
      batch_size = batch_size)
  
    print(res.history[&quot;loss&quot;])
  
    x_batch = []
    y_batch = []
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-04T10:37:20.250Z
Comments.Text: Can you please be more specific about "RNN cells can not cope with this simple task."? What do you mean by "cannot cope"? What's the problem you're having? How much data do you have? How exactly are you trying the RNNs? There are so many details missing from this question.
Comments.UserDisplayName: nbro
Tags: <recurrent-neural-networks><long-short-term-memory><gated-recurrent-unit><intuition>
Title: Is my intuition about RNN wrong?
UserDisplayName: Artod
ThreadText: # Is my intuition about RNN wrong?
  
  (Asked by: username_0 on 2022-01-02)
  
  Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs.
  Consider a task where you need to predict 0 if the previous input was 1. For example: x=[0,1,0,0], y=[1,1,0,1]. We train the model online, i.e. the number of timesteps the RNN sees at a time is equal to one (in Keras you can make an LSTM/GRU cell stateful so that it passes state (memory about previous inputs?) between batches). And the problem is that, in my tests, RNN cells can not cope with this simple task.
  I suspect that my intuition about RNN is built wrong. I would appreciate someone could help me with that.
  Test script:
  import numpy as np
  import tensorflow as tf
  from random import randrange
  
  batch_size = 1
  timesteps = 1
  size = 1
  units = 16
  shape = (timesteps, size)
  
  model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape = shape, batch_size = batch_size),
    tf.keras.layers.GRU(
      units,
      stateful = True,
      return_sequences = True
    )
  ])
  
  model.compile(optimizer='adam',
                loss='mean_squared_error')
  
  print(model.summary())
  
  SAME = False
  
  i = 0
  prevIsBlack = False
  x_batch, y_batch = [], []
  x_steps, y_steps = [], []
  
  while True:
    i += 1
  
    isBlack = randrange(4) == 2
  
    if SAME: prevIsBlack = isBlack 
  
    x_train = np.ones(size)   if isBlack else np.zeros(size)
    y_train = np.zeros(units) if prevIsBlack else np.ones(units)
  
    if not SAME: prevIsBlack = isBlack 
  
    x_steps.append(x_train)
    y_steps.append(y_train)
  
    if len(x_steps) < timesteps:
      continue
  
    x_batch.append(np.stack(x_steps))
    y_batch.append(np.stack(y_steps))
  
    x_steps, y_steps = [], []
  
    if len(x_batch) < batch_size:
      continue
  
    res = model.fit(np.stack(x_batch), np.stack(y_batch), 
      epochs = 1, 
      shuffle = False,
      batch_size = batch_size)
  
    print(res.history["loss"])
  
    x_batch = []
    y_batch = []
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, gated-recurrent-unit, intuition
  > **Comments:**
  > **username_1**: Can you please be more specific about "RNN cells can not cope with this simple task."? What do you mean by "cannot cope"? What's the problem you're having? How much data do you have? How exactly are you trying the RNNs? There are so many details missing from this question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-06. Score: 1
  
  I got this batch-based GRU to work:
  import numpy as np
  import tensorflow as tf
  
  units = 2
  seq_length = 32
  
  X = (np.random.rand(int(1e4), seq_length, 1) > 0.5).astype(float)
  Y = 1 - X
  
  model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=X.shape[1:]),
    tf.keras.layers.GRU(units, return_sequences=True),
    tf.keras.layers.Conv1D(1, 1, activation='linear')
  ])
  
  print(model.summary())
  
  model.compile(optimizer='adam', loss='mean_squared_error')
  history = model.fit(X, Y, batch_size=512, epochs=10, verbose=0, validation_split=0.1)
  
  for k in sorted(history.history.keys()):
      print('%15s: %s' % (k, str(np.array(history.history[k][:10]).round(3))))
  print()
  
  history = model.fit(X, Y, batch_size=512, epochs=40, verbose=0, validation_split=0.1)
  
  for k in sorted(history.history.keys()):
      print('%15s: %s' % (k, str(np.array(history.history[k][-10:]).round(3))))
  
  It trains for 10 epochs, prints logs, trains for 40 more epochs and prints logs of the last 10 epochs.
  Here is an example run:
  Model: "sequential_104"
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  gru_39 (GRU)                 (None, 32, 2)             30        
  _________________________________________________________________
  conv1d_68 (Conv1D)           (None, 32, 1)             3         
  =================================================================
  Total params: 33
  Trainable params: 33
  Non-trainable params: 0
  _________________________________________________________________
  
             loss: [1.137 0.97  0.832 0.718 0.625 0.546 0.479 0.422 0.372 0.329]
         val_loss: [1.041 0.891 0.767 0.665 0.58  0.508 0.447 0.394 0.348 0.307]
  
             loss: [0.026 0.023 0.021 0.019 0.017 0.016 0.014 0.013 0.011 0.01 ]
         val_loss: [0.024 0.022 0.02  0.018 0.016 0.015 0.013 0.012 0.011 0.009]
  
  Using 8 units helps it to converge faster, after 10 epochs the loss is about 0.1.
  I noticed that your network outputs the GRU's state directly as the predicted value.
  Maybe there is a problem since the output dimension doesn't match the target value? There might be some broadcasting going on, which hides the issue. <- edit: my earlier assumption, it isn't correct.
  My network's Conv1D converts the GRU's units dimensional output to a N x seq_length x 1 prediction. Adding it to your online version seems to fix it. Although the learning is quite unreliable, maybe due to the batch size of one.
  Edit: Aah after reading your code more carefully I noticed that the target predicted value isn't the previous value of the input (a single digit) but this:
  y_train = np.zeros(units) if prevIsBlack else np.ones(units)
  
  I don't know a neural network architecture in which we'd measure the loss directly on the RNN's output, it is only used by network to "remember" data or state from the past. But it is then processed further by subsequent layers. First of all the GRU's activation funtion is tanh, so it cannot even produce values outside of the range from -1 to 1.
  I hope this clears things up.
  
  
  ---
  
  