Body: <p>I'm currently trying to train a custom model with TensorFlow to detect 17 landmarks/keypoints on each of 2 hands shown in an image (fingertips, first knuckles, bottom knuckles, wrist, and palm), for 34 points (and therefore 68 total values to predict for x &amp; y). However, I cannot get the model to converge, with the output instead of being an array of points that are pretty much the same for every prediction.</p>
  <p>I started off with a dataset that has images like this:
  <a href="https://i.sstatic.net/tTY8K.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tTY8K.jpg" alt="enter image description here" /></a></p>
  <p>each annotated to have the red dots correlate to each keypoint. To expand the dataset to try to get a more robust model, I took photos of the hands with various backgrounds, angles, positions, poses, lighting conditions, reflectivity, etc, as exemplified by these further images:
  <a href="https://i.sstatic.net/qzQgU.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qzQgU.jpg" alt="enter image description here" /></a><a href="https://i.sstatic.net/IWry8.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IWry8.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/2tvmu.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2tvmu.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/Refoz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Refoz.png" alt="enter image description here" /></a> <a href="https://i.sstatic.net/N8gha.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/N8gha.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/ICp8k.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ICp8k.jpg" alt="enter image description here" /></a></p>
  <p>I have about 3000 images created now, with the landmarks stored inside a CSV as such:</p>
  <p><a href="https://i.sstatic.net/HVS6H.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HVS6H.png" alt="enter image description here" /></a></p>
  <p>I have a train-test split of .67 train .33 test, with the images randomly selected to each. I load the images with all 3 color channels and scale both the color values &amp; keypoint coordinates between 0 &amp; 1.</p>
  <p>I've tried a couple of different approaches, each involving a CNN. The first keeps the images as they are, and uses a neural network model built as such:</p>
  <pre><code>model = Sequential()
  
  model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu', input_shape = (225,400,3)))
  model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu'))
  model.add(MaxPooling2D(pool_size = (2,2), strides = 2))
  
  filters_convs = [(128, 2), (256, 3), (512, 3), (512,3)]
    
  for n_filters, n_convs in filters_convs:
    for _ in np.arange(n_convs):
      model.add(Conv2D(filters = n_filters, kernel_size = (3,3), padding = 'same', activation = 'relu'))
    model.add(MaxPooling2D(pool_size = (2,2), strides = 2))
  
  model.add(Flatten())
  model.add(Dense(128, activation=&quot;relu&quot;))
  model.add(Dense(96, activation=&quot;relu&quot;))
  model.add(Dense(72, activation=&quot;relu&quot;))
  model.add(Dense(68, activation=&quot;sigmoid&quot;))
  
  opt = Adam(learning_rate=.0001)
  model.compile(loss=&quot;mse&quot;, optimizer=opt, metrics=['mae'])
  print(model.summary())
  </code></pre>
  <p>I've modified the various hyperparameters, yet nothing seems to make any noticeable difference.</p>
  <p>The other thing I've tried is resizing the images to fit within a 224x224x3 array to use with a VGG-16 network, as such:</p>
  <pre><code>vgg = VGG16(weights=&quot;imagenet&quot;, include_top=False,
      input_tensor=Input(shape=(224, 224, 3)))
  vgg.trainable = False
  
  flatten = vgg.output
  flatten = Flatten()(flatten)
  
  points = Dense(256, activation=&quot;relu&quot;)(flatten)
  points = Dense(128, activation=&quot;relu&quot;)(points)
  points = Dense(96, activation=&quot;relu&quot;)(points)
  points = Dense(68, activation=&quot;sigmoid&quot;)(points)
  
  model = Model(inputs=vgg.input, outputs=points)
  
  opt = Adam(learning_rate=.0001)
  model.compile(loss=&quot;mse&quot;, optimizer=opt, metrics=['mae'])
  print(model.summary())
  </code></pre>
  <p>This model has similar results to the first. No matter what I seem to do, I seem to get the same results, in that my mse loss minimizes around .009, with an mae around .07, no matter how many epochs I run:
  <a href="https://i.sstatic.net/fgZri.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fgZri.png" alt="enter image description here" /></a></p>
  <p>Furthermore, when I run predictions based off the model it seems that the predicted output is basically the same for every image, with only a slight variation between each. It seems the model predicts an array of coordinates that looks somewhat like what a splayed hand might, in the general areas hands might be most likely to be found. A catch-all solution to minimize deviation as opposed to a custom solution for each image. These images illustrate this, with the green being predicted points, and the red being the actual points for the left hand:
  <a href="https://i.sstatic.net/pgWPH.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pgWPH.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/fLXvg.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fLXvg.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/W37Rs.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/W37Rs.jpg" alt="enter image description here" /></a> <a href="https://i.sstatic.net/iahQn.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iahQn.jpg" alt="enter image description here" /></a></p>
  <p>So, I was wondering what might be causing this, be it the model, the data, or both, because nothing I've tried with either modifying the model or augmenting the data seems to have done any good. I've even tried reducing the complexity to predict for one hand only, to predict a bounding box for each hand, and to predict a single keypoint, but no matter what I try, the results are pretty inaccurate.</p>
  <p>Thus, any suggestions for what I could do to help the model converge to create more accurate &amp; custom predictions for each image of hands it sees would be very greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-18T17:06:36.373Z
Comments.Text: Unless I am missing something it looks like your network is predicting the exact same thing. Look at face landmark detection examples.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-18T23:19:12.300Z
Comments.Text: @FourierFlux Thanks for the reply! You are not missing anything. You're right on the money. That's the issue. It seems to be predicting more of an average of all the values it's trained on rather than a custom solution for each image. And the models I built are actually based on facial landmark detection models repurposed for my data, so I've already spent a lot of time looking through such examples.
Comments.UserDisplayName: Sam Skinner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T01:27:27.130Z
Comments.Text: Look up the pytorch one which doesn't use dense and instead stays completely convolutional. It kind of makes sense to me that your dense layers are having issues.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T01:31:16.180Z
Comments.Text: I see. I'm not very familiar with pytorch, so I'll have to look into that. Do you have a link to or recommendation of any particular, model, guide, or paper in particular, or should it be easy enough to find & build?
Comments.UserDisplayName: Sam Skinner
Tags: <neural-networks><convolutional-neural-networks><computer-vision><tensorflow><convergence>
Title: Hand Landmark Detector Not Converging
UserDisplayName: Sam Skinner
ThreadText: # Hand Landmark Detector Not Converging
  
  (Asked by: username_0 on 2021-10-18)
  
  I'm currently trying to train a custom model with TensorFlow to detect 17 landmarks/keypoints on each of 2 hands shown in an image (fingertips, first knuckles, bottom knuckles, wrist, and palm), for 34 points (and therefore 68 total values to predict for x & y). However, I cannot get the model to converge, with the output instead of being an array of points that are pretty much the same for every prediction.
  I started off with a dataset that has images like this:
  
  each annotated to have the red dots correlate to each keypoint. To expand the dataset to try to get a more robust model, I took photos of the hands with various backgrounds, angles, positions, poses, lighting conditions, reflectivity, etc, as exemplified by these further images:
      
  I have about 3000 images created now, with the landmarks stored inside a CSV as such:
  
  I have a train-test split of .67 train .33 test, with the images randomly selected to each. I load the images with all 3 color channels and scale both the color values & keypoint coordinates between 0 & 1.
  I've tried a couple of different approaches, each involving a CNN. The first keeps the images as they are, and uses a neural network model built as such:
  model = Sequential()
  
  model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu', input_shape = (225,400,3)))
  model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu'))
  model.add(MaxPooling2D(pool_size = (2,2), strides = 2))
  
  filters_convs = [(128, 2), (256, 3), (512, 3), (512,3)]
    
  for n_filters, n_convs in filters_convs:
    for _ in np.arange(n_convs):
      model.add(Conv2D(filters = n_filters, kernel_size = (3,3), padding = 'same', activation = 'relu'))
    model.add(MaxPooling2D(pool_size = (2,2), strides = 2))
  
  model.add(Flatten())
  model.add(Dense(128, activation="relu"))
  model.add(Dense(96, activation="relu"))
  model.add(Dense(72, activation="relu"))
  model.add(Dense(68, activation="sigmoid"))
  
  opt = Adam(learning_rate=.0001)
  model.compile(loss="mse", optimizer=opt, metrics=['mae'])
  print(model.summary())
  
  I've modified the various hyperparameters, yet nothing seems to make any noticeable difference.
  The other thing I've tried is resizing the images to fit within a 224x224x3 array to use with a VGG-16 network, as such:
  vgg = VGG16(weights="imagenet", include_top=False,
      input_tensor=Input(shape=(224, 224, 3)))
  vgg.trainable = False
  
  flatten = vgg.output
  flatten = Flatten()(flatten)
  
  points = Dense(256, activation="relu")(flatten)
  points = Dense(128, activation="relu")(points)
  points = Dense(96, activation="relu")(points)
  points = Dense(68, activation="sigmoid")(points)
  
  model = Model(inputs=vgg.input, outputs=points)
  
  opt = Adam(learning_rate=.0001)
  model.compile(loss="mse", optimizer=opt, metrics=['mae'])
  print(model.summary())
  
  This model has similar results to the first. No matter what I seem to do, I seem to get the same results, in that my mse loss minimizes around .009, with an mae around .07, no matter how many epochs I run:
  
  Furthermore, when I run predictions based off the model it seems that the predicted output is basically the same for every image, with only a slight variation between each. It seems the model predicts an array of coordinates that looks somewhat like what a splayed hand might, in the general areas hands might be most likely to be found. A catch-all solution to minimize deviation as opposed to a custom solution for each image. These images illustrate this, with the green being predicted points, and the red being the actual points for the left hand:
     
  So, I was wondering what might be causing this, be it the model, the data, or both, because nothing I've tried with either modifying the model or augmenting the data seems to have done any good. I've even tried reducing the complexity to predict for one hand only, to predict a bounding box for each hand, and to predict a single keypoint, but no matter what I try, the results are pretty inaccurate.
  Thus, any suggestions for what I could do to help the model converge to create more accurate & custom predictions for each image of hands it sees would be very greatly appreciated.
  
  **Tags:** neural-networks, convolutional-neural-networks, computer-vision, tensorflow, convergence
  > **Comments:**
  > **username_1**: Unless I am missing something it looks like your network is predicting the exact same thing. Look at face landmark detection examples.  
  > **username_0**: @FourierFlux Thanks for the reply! You are not missing anything. You're right on the money. That's the issue. It seems to be predicting more of an average of all the values it's trained on rather than a custom solution for each image. And the models I built are actually based on facial landmark detection models repurposed for my data, so I've already spent a lot of time looking through such examples.  
  > **username_1**: Look up the pytorch one which doesn't use dense and instead stays completely convolutional. It kind of makes sense to me that your dense layers are having issues.  
  > **username_0**: I see. I'm not very familiar with pytorch, so I'll have to look into that. Do you have a link to or recommendation of any particular, model, guide, or paper in particular, or should it be easy enough to find & build?  
  
  ---
  
  ## 0 answers
  
  