Body: <p>I am approaching the implementation of the OpenPose algorithm for realtime human body pose estimation. </p>
  
  <p>According to the official paper <a href="https://arxiv.org/pdf/1812.08008.pdf" rel="nofollow noreferrer">OpenPose: Realtime Multi-Person 2D Pose
  Estimation using Part Affinity Fields</a>, <span class="math-container">$L$</span> and <span class="math-container">$S$</span> fields (body part maps, and part affinity fields) are estimated. These have the same size as the input image, and, according to the paper, these fields should be outputted at a given step in the forward pass (after a given number of <span class="math-container">$L$</span> stages, and <span class="math-container">$S$</span> stages), but, since before entering these stages the image is passed through the initial layers of the VGG-19 model, the spatial dimension is encoded and the features that finally enter the <span class="math-container">$L$</span> and <span class="math-container">$S$</span> stages have other dimensionality. </p>
  
  <p>All the network is convolutional, there's no FC layer at all. The VGG-19 part is the only one that contains MaxPooling layers, hence affecting the spatial relations and size of the receptive fields. </p>
  
  <p>My point is, after stage execution, I get tensors of shape <code>[batch_size, filter_number, 28, 28]</code>. The issue is that the paper is not stating how to decode this information into the <span class="math-container">$L$</span> and <span class="math-container">$S$</span> maps of size <span class="math-container">$224 \times 224$</span>.</p>
  
  <p>Following a traditional approach and decoding the final tensors with a linear net from, let's say, <span class="math-container">$15000 \rightarrow (224 * 224 * \text{ number of body parts }) + (224 * 224 * \text{ number of limbs } * 2)$</span>~A very huge number!, is out of question for any domestic computer, I presume I should have the least 128GbRAM installed, and is not the case. </p>
  
  <p>Another solution is to remove the max-pooling layers from the VGG-19 part, but then although the map size is preserved to <span class="math-container">$224$</span>, instead of <span class="math-container">$28$</span>, the huge amount of computations and values that need to be stored also lead to memory errors. </p>
  
  <p>So, the problem is, how can I get to a final output of <span class="math-container">$224 \times 224$</span> without FC layers, from a tensor of shape <code>[batch_size, bodyparts, 28, 28]</code>?</p>
  
  <p>Not an easy answer. I will check a TensorFlow implementation I have seen around to see how the problem was solved. </p>
  
  <p>Any parallel ideas are greatly welcome.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-22T11:12:28.353Z
Comments.Text: Sorry nbro, but that looked more to my english teacher corrections than a ML related answer. What was so necessary about your corrections?. I am not a native english speaker, but the overall meaning and sense of my question, is fully understandable I guess.
Comments.UserDisplayName: Chal.lo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-22T11:14:37.023Z
Comments.Text: BTW I think in other approaches the output of the S and L stages is upsampled to 224px to make it compatible with annotated images. So what I do is to downsample the annotations to the spatial dimensions of the output filters, (28,28) for the Loss function (MSE) to work. Now it seems to run correctly and there's no need to flatten or decode the results to compare results, which seems the right path to me. Any ideas?
Comments.UserDisplayName: Chal.lo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-24T00:25:04.740Z
Comments.Text: Please, ping me if you want me to see your messages, i.e. use `@nbro`. I didn't edit your post to offend you, but e.g. to improve the title so that it is more informative. I am sorry if you felt offended or upset. I was just trying to further improve your post. Unfortunately, I am not familiar with OpenPose, so I cannot help you now. I also didn't fully understand your description, probably because I am not familiar with OpenPose. Eventually, you could also ask this question at https://stats.stackexchange.com or https://datascience.stackexchange.com.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-24T08:51:20.137Z
Comments.Text: I didn't feel offended at all. I am aware my english expressivity is not fully accurate. Need to increase my lr. :D... no worries I fully understand...
Comments.UserDisplayName: Chal.lo
Tags: <neural-networks><deep-learning><papers><implementation><pose-estimation>
Title: How can I get to a final output of shape $224 \times 224$, without FC layers, from a tensor of specific shape, in OpenPose?
UserDisplayName: Chal.lo
ThreadText: # How can I get to a final output of shape $224 \times 224$, without FC layers, from a tensor of specific shape, in OpenPose?
  
  (Asked by: username_0 on 2020-03-21)
  
  I am approaching the implementation of the OpenPose algorithm for realtime human body pose estimation.
  
  According to the official paper [OpenPose: Realtime Multi-Person 2D Pose
  Estimation using Part Affinity Fields](https://arxiv.org/pdf/1812.08008.pdf), $L$ and $S$ fields (body part maps, and part affinity fields) are estimated. These have the same size as the input image, and, according to the paper, these fields should be outputted at a given step in the forward pass (after a given number of $L$ stages, and $S$ stages), but, since before entering these stages the image is passed through the initial layers of the VGG-19 model, the spatial dimension is encoded and the features that finally enter the $L$ and $S$ stages have other dimensionality.
  
  All the network is convolutional, there's no FC layer at all. The VGG-19 part is the only one that contains MaxPooling layers, hence affecting the spatial relations and size of the receptive fields.
  
  My point is, after stage execution, I get tensors of shape `[batch_size, filter_number, 28, 28]`. The issue is that the paper is not stating how to decode this information into the $L$ and $S$ maps of size $224 \times 224$.
  
  Following a traditional approach and decoding the final tensors with a linear net from, let's say, $15000 \rightarrow (224 \* 224 \* \text{ number of body parts }) + (224 \* 224 \* \text{ number of limbs } \* 2)$~A very huge number!, is out of question for any domestic computer, I presume I should have the least 128GbRAM installed, and is not the case.
  
  Another solution is to remove the max-pooling layers from the VGG-19 part, but then although the map size is preserved to $224$, instead of $28$, the huge amount of computations and values that need to be stored also lead to memory errors.
  
  So, the problem is, how can I get to a final output of $224 \times 224$ without FC layers, from a tensor of shape `[batch_size, bodyparts, 28, 28]`?
  
  Not an easy answer. I will check a TensorFlow implementation I have seen around to see how the problem was solved.
  
  Any parallel ideas are greatly welcome.
  
  **Tags:** neural-networks, deep-learning, papers, implementation, pose-estimation
  
  > **Comments:**
  > **username_0**: Sorry username_1, but that looked more to my english teacher corrections than a ML related answer. What was so necessary about your corrections?. I am not a native english speaker, but the overall meaning and sense of my question, is fully understandable I guess.  
  > **username_0**: BTW I think in other approaches the output of the S and L stages is upsampled to 224px to make it compatible with annotated images. So what I do is to downsample the annotations to the spatial dimensions of the output filters, (28,28) for the Loss function (MSE) to work. Now it seems to run correctly and there's no need to flatten or decode the results to compare results, which seems the right path to me. Any ideas?  
  > **username_1**: Please, ping me if you want me to see your messages, i.e. use `@username_1`. I didn't edit your post to offend you, but e.g. to improve the title so that it is more informative. I am sorry if you felt offended or upset. I was just trying to further improve your post. Unfortunately, I am not familiar with OpenPose, so I cannot help you now. I also didn't fully understand your description, probably because I am not familiar with OpenPose. Eventually, you could also ask this question at https://stats.stackexchange.com or https://datascience.stackexchange.com.  
  > **username_0**: I didn't feel offended at all. I am aware my english expressivity is not fully accurate. Need to increase my lr. :D... no worries I fully understand...  
  
  ---
  
  ## 0 answers
  
  