Answers.Body: <p>your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-02T05:42:13.643Z
Answers.LastActivityDate: 2018-06-02T05:42:13.643Z
Answers.UserDisplayName: thecomplexitytheorist
Body: <p>I am currently trying to solve a regression problem using neural networks. I want to detect movement patterns in images over time (video) and output a continuous value. 
  During the training process I noticed a strange behaviour for the validation loss curve and I was wondering if anyone has noticed this kind of <strong>periodic pattern</strong> on some of their own work. <strong>What might cause this?</strong> </p>
  
  <p>The model looks like the following:  </p>
  
  <pre><code>- TimeDistributed(Conv2D(32, (3,3)))
  - TimeDistributed(Conv2D(16, (3,3)))
  - TimeDistributed(Flatten())
  - GRU(64, stateful=True)
  - Dropout(0.5)
  - Dense(64, activation='relu')
  - Dense(1)
  </code></pre>
  
  <p>I trained the model using the mean squared error as the loss function, a batch size of 1 and the AdamOptimizer with an initial learning rate of 10^(-6). Obviously, the loss curve for the training data is not very good, but I am currently just wondering about the pattern of the val_loss. The plots below represent the loss of 65 epochs.</p>
  
  <p>Thanks!</p>
  
  <p><a href="https://i.sstatic.net/ebRvf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ebRvf.png" alt="Validation Loss"></a></p>
  
  <p><a href="https://i.sstatic.net/HK88g.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HK88g.png" alt="Training Loss"></a></p>
  
  <p><strong>Edit:</strong>
  The way I try to solve my task relies on a sliding window approach where I try to predict a continuous value for the next second based on the last 20 seconds (400 frames) of the time-series input data. But I don't think this information is needed to solve my initial question since the periodic patterns appear over several epochs (one "peak" for about every 15 epochs) which is strange. Although the stateful-version of the GRU is used (btw: using TensorFlow and Keras), the internal state of the GRU is reset after every epoch to maintain a clean start. The stateful keyword is used to indicate a dependency between batches.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-18T21:20:39.120Z
Comments.Text: To add some more clarity, can you explain more about how you set up your dataset and training pipeline? Perhaps there's a link to a tutorial or a source code you were following?
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-10T20:10:35.703Z
Comments.Text: Are you sure that you're going over all your dataset in one epoch? The fact that the loss also has these spikes suggests that you might be seeing new training data you haven't trained on before?
Comments.UserDisplayName: Raphael Lopez Kaufman
Tags: <deep-learning><training><keras>
Title: Periodic Pattern in Validation Loss Curve
UserDisplayName: Unknown User
ThreadText: # Periodic Pattern in Validation Loss Curve
  
  (Asked by: username_0 on 2018-03-01)
  
  I am currently trying to solve a regression problem using neural networks. I want to detect movement patterns in images over time (video) and output a continuous value. 
  During the training process I noticed a strange behaviour for the validation loss curve and I was wondering if anyone has noticed this kind of periodic pattern on some of their own work. What might cause this? 
  The model looks like the following:  
  - TimeDistributed(Conv2D(32, (3,3)))
  - TimeDistributed(Conv2D(16, (3,3)))
  - TimeDistributed(Flatten())
  - GRU(64, stateful=True)
  - Dropout(0.5)
  - Dense(64, activation='relu')
  - Dense(1)
  
  I trained the model using the mean squared error as the loss function, a batch size of 1 and the AdamOptimizer with an initial learning rate of 10^(-6). Obviously, the loss curve for the training data is not very good, but I am currently just wondering about the pattern of the val_loss. The plots below represent the loss of 65 epochs.
  Thanks!
  
  
  Edit:
  The way I try to solve my task relies on a sliding window approach where I try to predict a continuous value for the next second based on the last 20 seconds (400 frames) of the time-series input data. But I don't think this information is needed to solve my initial question since the periodic patterns appear over several epochs (one "peak" for about every 15 epochs) which is strange. Although the stateful-version of the GRU is used (btw: using TensorFlow and Keras), the internal state of the GRU is reset after every epoch to maintain a clean start. The stateful keyword is used to indicate a dependency between batches.
  
  **Tags:** deep-learning, training, keras
  > **Comments:**
  > **username_1**: To add some more clarity, can you explain more about how you set up your dataset and training pipeline? Perhaps there's a link to a tutorial or a source code you were following?  
  > **username_2**: Are you sure that you're going over all your dataset in one epoch? The fact that the loss also has these spikes suggests that you might be seeing new training data you haven't trained on before?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-06-02. Score: 0
  
  your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.
  
  
  ---
  
  