Answers.Body: <p>I agree with <a href="https://ai.stackexchange.com/a/43686/52211">the answer by @vl_knd</a> and want to add my two cents:</p>
  <p>This problem does sound a lot like a special case of image segmentation. You want an image where each pixel gets assigned a class. In your case, most pixels will be class 0 and some individual pixels will be class 1, that's where the points are. A very similar field of research is <strong>2D pose estimation</strong>. Given an image, you want to map a skeleton onto that image that indicates the pose of a depicted person. This is mostly mapping 2D points onto the joints of a person. There exists work for both the ideas you mentioned (regression of <span class="math-container">$x,y$</span> and prediction a heat-map representation of the <span class="math-container">$x,y$</span>-points on an image) <a href="https://www.fubin.org/research/Human_Pose_Estimation/Human_Pose_Estimation.html" rel="nofollow noreferrer">Here is a reading list of such papers</a>. And <a href="https://www.yangangwang.com/papers/WANG-SRH-2019-11.pdf" rel="nofollow noreferrer"><strong>this paper</strong></a> <strong>predicts the <span class="math-container">$x, y$</span> pixels in the form of heat maps</strong>.</p>
  <p>You might also find some ideas in papers that tackle high-precision segmentation problems. The medical domain might be interesting (i.e. blood vessel segmentation).</p>
  <p>When it comes to the loss function, it should probably be a bit forgiving. When you want a pixel <span class="math-container">$(x,y)$</span> to be classified as 1, neighboring pixels likely have some ambiguity as to whether they should be 1 as well. What can come in handy is smooth labeling. You could for example center a Gaussian at <span class="math-container">$(x,y)$</span> with some variance to include neighboring pixels and provide a smoother training signal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-05T08:44:17.093Z
Answers.LastActivityDate: 2024-02-05T08:57:32.557Z
Answers.LastEditDate: 2024-02-05T08:57:32.557Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>It looks like you are thinking about something like <a href="https://en.wikipedia.org/wiki/U-Net" rel="nofollow noreferrer">UNET</a> architecture.</p>
  <p>The final layer to your toy problem with a &quot;dot&quot; on the nose can be modelled as a convolution layer from last convolution layer of your network of a shape [BatchSize, Channels, Width, Hight] to the &quot;output&quot; layer of size [BatchSize, 1, Width, Hight].</p>
  <p>You may have a dimensionality problem, since the width and hight of the output image can differ from the input. This can happened if you decide to, for example, perform pooling in your architecture, luckily there are a lot of <a href="https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html" rel="nofollow noreferrer">upsampling methods</a>. You also can use architecture of CNN, that does not change image's size, like <a href="https://en.wikipedia.org/wiki/U-Net" rel="nofollow noreferrer">UNET</a>, <a href="https://arxiv.org/abs/1506.02640" rel="nofollow noreferrer">YOLO</a>. I recommend you to look at the problem of <a href="https://neptune.ai/blog/image-segmentation" rel="nofollow noreferrer">image segmentation</a>, since it looks very similar to the example given by you.</p>
  <p>As of your second point. If you would want to predict a mask of onces and zeros you can use sigmoid function for each activation ( pixel ) of last convolution layer to get the probability as an output and then train the network by minimising <a href="https://en.wikipedia.org/wiki/Cross-entropy" rel="nofollow noreferrer">cross entropy</a> between predicted probabilities and true &quot;mask&quot; of an image.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-05T00:03:50.160Z
Answers.LastActivityDate: 2024-02-05T00:03:50.160Z
Answers.UserDisplayName: vl_knd
Body: <p>I'm currently studying CNNs and I had the idea of building a model without a fully connected layer at the end. I think this could be beneficial, if one can somehow model the desired outputs as a matrix. A basic problem solvable in this way is the following: let's say we want to place a dot on the nose of a person in an image. The basic approach is</p>
  <pre><code>img -&gt; [conv layer] -&gt; [pooling] -&gt; [...] -&gt; [fully connected layer] -&gt; pred_x, pred_y
  </code></pre>
  <p>where <code>pred_x</code> and <code>pred_y</code> are the two nodes in the final layer, after the fully connected one, and represents the coordinates of the dot.</p>
  <p>Every coordinate <code>x, y</code> can be represented with a matrix of size <code>image_width x image_height</code> with all zeros except a <code>1</code> in the position <code>int(x), int(y)</code>. This means that it might be possible to have a CNN like the following:</p>
  <pre><code>img -&gt; [conv layer] -&gt; [pooling] -&gt; [conv layer] -&gt; [...] -&gt; act_map
  </code></pre>
  <p>where <code>act_map</code> is the activation map resulting from the last convolutional layer. Given <code>act_map</code> we can extract <code>pred_x, pred_y</code> by getting the two indexes for which <code>act_map</code> is maximum.</p>
  <p>Maybe this doesn't make sense with a single output, but imagine a problem where we want to predict the position of <code>n</code> dots on an image. Then we could simply have <code>n</code> activation maps at the end.</p>
  <p>I have a few questions:</p>
  <ol>
  <li>where can I find more resources on this topic? I'm sure I'm not the first one to have thought about this</li>
  <li>could this really work? I don't really know how to create a loss function</li>
  </ol>
  <p>Thanks</p>
  
Tags: <neural-networks><convolutional-neural-networks><convolution>
Title: What's the best criterion for evaluating activation maps in a CNN?
UserDisplayName: Pietro
ThreadText: # What's the best criterion for evaluating activation maps in a CNN?
  
  (Asked by: username_0 on 2024-02-01)
  
  I'm currently studying CNNs and I had the idea of building a model without a fully connected layer at the end. I think this could be beneficial, if one can somehow model the desired outputs as a matrix. A basic problem solvable in this way is the following: let's say we want to place a dot on the nose of a person in an image. The basic approach is
  img -> [conv layer] -> [pooling] -> [...] -> [fully connected layer] -> pred_x, pred_y
  
  where pred_x and pred_y are the two nodes in the final layer, after the fully connected one, and represents the coordinates of the dot.
  Every coordinate x, y can be represented with a matrix of size image_width x image_height with all zeros except a 1 in the position int(x), int(y). This means that it might be possible to have a CNN like the following:
  img -> [conv layer] -> [pooling] -> [conv layer] -> [...] -> act_map
  
  where act_map is the activation map resulting from the last convolutional layer. Given act_map we can extract pred_x, pred_y by getting the two indexes for which act_map is maximum.
  Maybe this doesn't make sense with a single output, but imagine a problem where we want to predict the position of n dots on an image. Then we could simply have n activation maps at the end.
  I have a few questions:
  
  where can I find more resources on this topic? I'm sure I'm not the first one to have thought about this
  could this really work? I don't really know how to create a loss function
  
  Thanks
  
  **Tags:** neural-networks, convolutional-neural-networks, convolution
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-02-05. Score: 3 | Accepted answer
  
  It looks like you are thinking about something like [UNET](https://en.wikipedia.org/wiki/U-Net) architecture.
  
  The final layer to your toy problem with a "dot" on the nose can be modelled as a convolution layer from last convolution layer of your network of a shape [BatchSize, Channels, Width, Hight] to the "output" layer of size [BatchSize, 1, Width, Hight].
  
  You may have a dimensionality problem, since the width and hight of the output image can differ from the input. This can happened if you decide to, for example, perform pooling in your architecture, luckily there are a lot of [upsampling methods](https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html). You also can use architecture of CNN, that does not change image's size, like [UNET](https://en.wikipedia.org/wiki/U-Net), [YOLO](https://arxiv.org/abs/1506.02640). I recommend you to look at the problem of [image segmentation](https://neptune.ai/blog/image-segmentation), since it looks very similar to the example given by you.
  
  As of your second point. If you would want to predict a mask of onces and zeros you can use sigmoid function for each activation ( pixel ) of last convolution layer to get the probability as an output and then train the network by minimising [cross entropy](https://en.wikipedia.org/wiki/Cross-entropy) between predicted probabilities and true "mask" of an image.
  
  
  ---
  
  ### Answer by username_1 on 2024-02-05. Score: 2
  
  I agree with [the answer by @vl\_knd](https://ai.stackexchange.com/a/43686/52211) and want to add my two cents:
  
  This problem does sound a lot like a special case of image segmentation. You want an image where each pixel gets assigned a class. In your case, most pixels will be class 0 and some individual pixels will be class 1, that's where the points are. A very similar field of research is **2D pose estimation**. Given an image, you want to map a skeleton onto that image that indicates the pose of a depicted person. This is mostly mapping 2D points onto the joints of a person. There exists work for both the ideas you mentioned (regression of $x,y$ and prediction a heat-map representation of the $x,y$-points on an image) [Here is a reading list of such papers](https://www.fubin.org/research/Human_Pose_Estimation/Human_Pose_Estimation.html). And [**this paper**](https://www.yangangwang.com/papers/WANG-SRH-2019-11.pdf) **predicts the $x, y$ pixels in the form of heat maps**.
  
  You might also find some ideas in papers that tackle high-precision segmentation problems. The medical domain might be interesting (i.e. blood vessel segmentation).
  
  When it comes to the loss function, it should probably be a bit forgiving. When you want a pixel $(x,y)$ to be classified as 1, neighboring pixels likely have some ambiguity as to whether they should be 1 as well. What can come in handy is smooth labeling. You could for example center a Gaussian at $(x,y)$ with some variance to include neighboring pixels and provide a smoother training signal.
  
  
  ---
  
  