Answers.Body: <p>In the end, this problem turned out to be largely a matter of the gradient descent falling into local minima.</p>
  <p>For those reading for posterity, one of the issues in ML that is difficult to work around is that we cannot intuitively choose reasonable initial values for the weights, biases, and kernels (in the CNN).  As a result, we typically allow them to initialize randomly.  This can present some challenges.</p>
  <p>One of the biggest challenges is that when you start from a random starting point, it's difficult to tell someone how to completely replicate your experiments.  This isn't terribly important in the end since you can provide them with the saved parameters from your trained model.  However, this can also lead to networks that appear to be &quot;bad&quot; that are in fact perfectly fine.</p>
  <p>In this case, I had spent much of the time initializing the CNN with a uniform initializer (not present in the code above).  I will sometimes use a random seed or some other function to generate initial values so that I can better improve networks through genetic search tools.</p>
  <p>It seems that the uniform initializers combined with the various network iterations and this particular data lead to absolutely abysmal training performance and non-convergence.</p>
  <p>When I ran the network as above with random initializations and one or two tweaks, it converged well.  Some training iterations will pin one of the sides of the bounding box at the edge, some will never converge, but I've managed to successfully train several that are in the 96-98% accuracy range for the bounding boxes in my test set of 20000, so all is well!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T01:56:40.093Z
Answers.Text: When you say "When I ran the network as above with random initializations and one or two tweaks", which specific initialization technique did you use in the end? Note that uniform initialization is also random. What tweaks did you use?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T12:51:57.097Z
Answers.Text: I had been using `hm_uniform`.  I simply allowed the default initialization to take place.  I need to plug this into one of my genetic search algorithms after finding a good seed that produces good results to better refine the model.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T00:35:30.483Z
Answers.LastActivityDate: 2021-01-21T00:35:30.483Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I've solved many problems with neural networks, but rarely work with images.  I have about 18 hours into creating a bounding box regression network and it continues to utterly fail.  With some loss functions it will claim 80% accuracy during training and validation (with a truly massive loss on both) but testing the predictions reveals a bounding box that only moves one or two pixels in any given direction and seems to totally ignore the data.  I've now implemented a form of IoU loss, but find that IoU is pinned at zero... which is obviously true based on the outputs after training. :). I'd like someone to look this over and give me some advice on how to proceed next.</p>
  <h2>What I Have</h2>
  <p>I am generating 40000 examples of 200x100x3 images with a single letter randomly placed in each.  Simultaneously I am generating the ground truth bounding boxes for each training sample.  I have thoroughly validated that this all works and the data is correct.</p>
  <h2>What I Do To It</h2>
  <p>I am then transforming the 200x100x3 images down to greyscale to produce a 200x100x1 image.  The images are then normalized and the bounding boxes are scaled to fall between 0 and 1.  In simplified form, this happens:</p>
  <pre><code>x_train_normalized = (x_data - 127.5) / 127.5
  y_train_scaled = boxes[:TRAIN]/[WIDTH,HEIGHT,WIDTH,HEIGHT]
  </code></pre>
  <p>I've been through this data carefully, even reconstituting images and bounding boxes from it.  This is definitely working.</p>
  <h2>Training</h2>
  <p>To train, after trying <code>mse</code> and many others, all of which fail equally badly, I have implemented a simple custom IOU loss function.  It actually returns <code>-ln(IoU)</code>.  I made this change based on a paper since the loss was (oddly?) pinned at zero over multiple epochs.</p>
  <p>(Loss function:)</p>
  <pre><code>import tensorflow.keras.backend as kb
  def iou_loss(y_actual,y_pred):
      b1 = y_actual
      b2 = y_pred
  #    tf.print(b1)
  #    tf.print(b2)
      zero = tf.convert_to_tensor(0.0, b1.dtype)
      b1_ymin, b1_xmin, b1_ymax, b1_xmax = tf.unstack(b1, 4, axis=-1)
      b2_ymin, b2_xmin, b2_ymax, b2_xmax = tf.unstack(b2, 4, axis=-1)
      b1_width = tf.maximum(zero, b1_xmax - b1_xmin)
      b1_height = tf.maximum(zero, b1_ymax - b1_ymin)
      b2_width = tf.maximum(zero, b2_xmax - b2_xmin)
      b2_height = tf.maximum(zero, b2_ymax - b2_ymin)
      b1_area = b1_width * b1_height
      b2_area = b2_width * b2_height
  
      intersect_ymin = tf.maximum(b1_ymin, b2_ymin)
      intersect_xmin = tf.maximum(b1_xmin, b2_xmin)
      intersect_ymax = tf.minimum(b1_ymax, b2_ymax)
      intersect_xmax = tf.minimum(b1_xmax, b2_xmax)
      intersect_width = tf.maximum(zero, intersect_xmax - intersect_xmin)
      intersect_height = tf.maximum(zero, intersect_ymax - intersect_ymin)
      intersect_area = intersect_width * intersect_height
  
      union_area = b1_area + b2_area - intersect_area
      iou = -1 * tf.math.log(tf.math.divide_no_nan(intersect_area, union_area))
      return iou
  </code></pre>
  <h3>The Network</h3>
  <p>This has been through many, many iterations.  As I said, I've solved many other problems with NNs... This is the first one to get me completely stuck.  At this point, the network is dramatically stripped down but continues to fail to train at all:</p>
  <pre><code>import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras import layers, optimizers
  
  tf.keras.backend.set_floatx('float32') # Use Float32s for everything
  
  input_shape = x_train_normalized.shape[-3:]
  model = keras.Sequential()
  model.add(layers.Conv2D(4, 16, activation = tf.keras.layers.LeakyReLU(alpha=0.2), input_shape=input_shape))
  model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=(2, 2)))
  model.add(layers.Dropout(0.2))
  model.add(layers.Flatten())
  model.add(layers.Dense(200, activation = tf.keras.layers.LeakyReLU(alpha=0.2)))
  model.add(layers.Dense(64, activation=tf.keras.layers.LeakyReLU(alpha=0.2)))
  model.add(layers.Dense(4, activation=&quot;sigmoid&quot;))
  
  model.compile(loss = iou_loss, optimizer = &quot;adadelta&quot;, metrics=['accuracy'])
  history = model.fit(x_train_normalized, y_train_scaled, epochs=8, batch_size=100, validation_split=0.4)
  </code></pre>
  <p>All pointers are welcome!  In the meantime I'm implementing a center point loss function to see if that helps at all.</p>
  
Tags: <tensorflow><object-detection><weights-initialization><bounding-box>
Title: Bounding Box Regression - An Adventure in Failure
UserDisplayName: David Hoelzer
ThreadText: # Bounding Box Regression - An Adventure in Failure
  
  (Asked by: username_0 on 2021-01-19)
  
  I've solved many problems with neural networks, but rarely work with images.  I have about 18 hours into creating a bounding box regression network and it continues to utterly fail.  With some loss functions it will claim 80% accuracy during training and validation (with a truly massive loss on both) but testing the predictions reveals a bounding box that only moves one or two pixels in any given direction and seems to totally ignore the data.  I've now implemented a form of IoU loss, but find that IoU is pinned at zero... which is obviously true based on the outputs after training. :). I'd like someone to look this over and give me some advice on how to proceed next.
  What I Have
  I am generating 40000 examples of 200x100x3 images with a single letter randomly placed in each.  Simultaneously I am generating the ground truth bounding boxes for each training sample.  I have thoroughly validated that this all works and the data is correct.
  What I Do To It
  I am then transforming the 200x100x3 images down to greyscale to produce a 200x100x1 image.  The images are then normalized and the bounding boxes are scaled to fall between 0 and 1.  In simplified form, this happens:
  x_train_normalized = (x_data - 127.5) / 127.5
  y_train_scaled = boxes[:TRAIN]/[WIDTH,HEIGHT,WIDTH,HEIGHT]
  
  I've been through this data carefully, even reconstituting images and bounding boxes from it.  This is definitely working.
  Training
  To train, after trying mse and many others, all of which fail equally badly, I have implemented a simple custom IOU loss function.  It actually returns -ln(IoU).  I made this change based on a paper since the loss was (oddly?) pinned at zero over multiple epochs.
  (Loss function:)
  import tensorflow.keras.backend as kb
  def iou_loss(y_actual,y_pred):
      b1 = y_actual
      b2 = y_pred
  #    tf.print(b1)
  #    tf.print(b2)
      zero = tf.convert_to_tensor(0.0, b1.dtype)
      b1_ymin, b1_xmin, b1_ymax, b1_xmax = tf.unstack(b1, 4, axis=-1)
      b2_ymin, b2_xmin, b2_ymax, b2_xmax = tf.unstack(b2, 4, axis=-1)
      b1_width = tf.maximum(zero, b1_xmax - b1_xmin)
      b1_height = tf.maximum(zero, b1_ymax - b1_ymin)
      b2_width = tf.maximum(zero, b2_xmax - b2_xmin)
      b2_height = tf.maximum(zero, b2_ymax - b2_ymin)
      b1_area = b1_width * b1_height
      b2_area = b2_width * b2_height
  
      intersect_ymin = tf.maximum(b1_ymin, b2_ymin)
      intersect_xmin = tf.maximum(b1_xmin, b2_xmin)
      intersect_ymax = tf.minimum(b1_ymax, b2_ymax)
      intersect_xmax = tf.minimum(b1_xmax, b2_xmax)
      intersect_width = tf.maximum(zero, intersect_xmax - intersect_xmin)
      intersect_height = tf.maximum(zero, intersect_ymax - intersect_ymin)
      intersect_area = intersect_width * intersect_height
  
      union_area = b1_area + b2_area - intersect_area
      iou = -1 * tf.math.log(tf.math.divide_no_nan(intersect_area, union_area))
      return iou
  
  The Network
  This has been through many, many iterations.  As I said, I've solved many other problems with NNs... This is the first one to get me completely stuck.  At this point, the network is dramatically stripped down but continues to fail to train at all:
  import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras import layers, optimizers
  
  tf.keras.backend.set_floatx('float32') # Use Float32s for everything
  
  input_shape = x_train_normalized.shape[-3:]
  model = keras.Sequential()
  model.add(layers.Conv2D(4, 16, activation = tf.keras.layers.LeakyReLU(alpha=0.2), input_shape=input_shape))
  model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=(2, 2)))
  model.add(layers.Dropout(0.2))
  model.add(layers.Flatten())
  model.add(layers.Dense(200, activation = tf.keras.layers.LeakyReLU(alpha=0.2)))
  model.add(layers.Dense(64, activation=tf.keras.layers.LeakyReLU(alpha=0.2)))
  model.add(layers.Dense(4, activation="sigmoid"))
  
  model.compile(loss = iou_loss, optimizer = "adadelta", metrics=['accuracy'])
  history = model.fit(x_train_normalized, y_train_scaled, epochs=8, batch_size=100, validation_split=0.4)
  
  All pointers are welcome!  In the meantime I'm implementing a center point loss function to see if that helps at all.
  
  **Tags:** tensorflow, object-detection, weights-initialization, bounding-box
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-01-21. Score: 0 | Accepted answer
  
  In the end, this problem turned out to be largely a matter of the gradient descent falling into local minima.
  
  For those reading for posterity, one of the issues in ML that is difficult to work around is that we cannot intuitively choose reasonable initial values for the weights, biases, and kernels (in the CNN). As a result, we typically allow them to initialize randomly. This can present some challenges.
  
  One of the biggest challenges is that when you start from a random starting point, it's difficult to tell someone how to completely replicate your experiments. This isn't terribly important in the end since you can provide them with the saved parameters from your trained model. However, this can also lead to networks that appear to be "bad" that are in fact perfectly fine.
  
  In this case, I had spent much of the time initializing the CNN with a uniform initializer (not present in the code above). I will sometimes use a random seed or some other function to generate initial values so that I can better improve networks through genetic search tools.
  
  It seems that the uniform initializers combined with the various network iterations and this particular data lead to absolutely abysmal training performance and non-convergence.
  
  When I ran the network as above with random initializations and one or two tweaks, it converged well. Some training iterations will pin one of the sides of the bounding box at the edge, some will never converge, but I've managed to successfully train several that are in the 96-98% accuracy range for the bounding boxes in my test set of 20000, so all is well!
  
  > **Comments:**
  > **username_1**: When you say "When I ran the network as above with random initializations and one or two tweaks", which specific initialization technique did you use in the end? Note that uniform initialization is also random. What tweaks did you use?  
  > **username_0**: I had been using `hm_uniform`.  I simply allowed the default initialization to take place.  I need to plug this into one of my genetic search algorithms after finding a good seed that produces good results to better refine the model.  
  
  ---
  
  