Answers.Body: <p>The problem was the dimensions of the <code>logit_length</code> argument to <code>tf.nn.ctc_loss</code> was incorrect.</p>
  <p>It was this:</p>
  <pre><code>tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])
  </code></pre>
  <p>But it should have been</p>
  <pre><code>tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T15:40:05.830Z
Answers.LastActivityDate: 2021-01-17T15:40:05.830Z
Answers.UserDisplayName: Cameron Martin
Body: <p>I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy the current input element at each time step. Moreover, I have made the length of the label the same as the length of the input sequence and no adjacent elements in the label sequence the same so that both CTC loss and categorical cross-entropy loss can be used. I found that when using categorical cross-entropy loss the model very quickly converges, whereas when using CTC loss it gets nowhere.</p>
  <p>I have <a href="https://colab.research.google.com/drive/1gMqWlq80i5MA9BhOCfbpVLAg5BQSg0JO?usp=sharing" rel="nofollow noreferrer">uploaded by minimal example to colab</a>. Does anyone know why CTC loss is not working in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T21:34:13.293Z
Comments.Text: Could you please provide the mathematical formula of the CTC loss that you implemented, to have more context?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T14:56:15.153Z
Comments.Text: I did not implement the loss function myself, but rather am using a built-in function from tensorflow, [`tf.nn.ctc_loss`](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss). The mathematical formulation of this is available in the paper [_Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks_](http://www.cs.toronto.edu/~graves/icml_2006.pdf).
Comments.UserDisplayName: Cameron Martin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T17:18:22.450Z
Comments.Text: Thanks! I would like to note that programming issues are off-topic here. So, next time, if you think your problem is a bug in your code and not a conceptual issue, then you should ask your question on Stack Overflow. Here, we focus on the theoretical, philosophical, and social aspects of Artificial Intelligence. Take a look at https://ai.stackexchange.com/help/on-topic.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><tensorflow><long-short-term-memory><ctc-loss>
Title: Why won't my model train with CTC loss?
UserDisplayName: Cameron Martin
ThreadText: # Why won't my model train with CTC loss?
  
  (Asked by: username_0 on 2021-01-16)
  
  I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy the current input element at each time step. Moreover, I have made the length of the label the same as the length of the input sequence and no adjacent elements in the label sequence the same so that both CTC loss and categorical cross-entropy loss can be used. I found that when using categorical cross-entropy loss the model very quickly converges, whereas when using CTC loss it gets nowhere.
  
  I have [uploaded by minimal example to colab](https://colab.research.google.com/drive/1gMqWlq80i5MA9BhOCfbpVLAg5BQSg0JO?usp=sharing). Does anyone know why CTC loss is not working in this case?
  
  **Tags:** neural-networks, deep-learning, tensorflow, long-short-term-memory, ctc-loss
  
  > **Comments:**
  > **username_1**: Could you please provide the mathematical formula of the CTC loss that you implemented, to have more context?  
  > **username_0**: I did not implement the loss function myself, but rather am using a built-in function from tensorflow, [`tf.nn.ctc_loss`](https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss). The mathematical formulation of this is available in the paper [_Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks_](http://www.cs.toronto.edu/~graves/icml_2006.pdf).  
  > **username_1**: Thanks! I would like to note that programming issues are off-topic here. So, next time, if you think your problem is a bug in your code and not a conceptual issue, then you should ask your question on Stack Overflow. Here, we focus on the theoretical, philosophical, and social aspects of Artificial Intelligence. Take a look at https://ai.stackexchange.com/help/on-topic.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-01-17. Score: 0
  
  The problem was the dimensions of the `logit_length` argument to `tf.nn.ctc_loss` was incorrect.
  
  It was this:
  
  ```
  tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])
  
  ```
  
  But it should have been
  
  ```
  tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])
  
  ```
  
  
  ---
  
  