Answers.Body: <p>This is a while ago, but still this problem might occur to someone.
  I encountered the same problem and found that the reason was how the resulting tensors are transformed to images. It seams that pixels with value 1.0 are shown as white while pixels with values bigger than 1.0 are interpreted as being black sometimes.
  So adding (in PyTorch)</p>
  <pre><code>indices = tensor &gt; 1.0
  tensor[indices] = 1.0 
  </code></pre>
  <p>before converting to the image in the visualization method fixed the problem for me.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-29T16:20:13.667Z
Answers.Text: Do you remember which layer in neural network? Was it some tensor in the last layer? Also which Super Resolution method (list at https://github.com/LoSealL/VideoSuperResolution#network-list-and-reference-updating)?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T17:39:02.450Z
Answers.Text: It was not in a layer at all in that sense. The Network outputs values that are mostly between 1 and 0. But sometimes these values are slightly over 1 or under 0 if no activation function is used after the last layer (e. g. no sigmoid) . So the network gives a value whiter than white so to say. Independent of the super resolution method this might always happen without activation function. However, some visualization packets show values bigger than 1 as black. So you might fix that by clipping the values between 0 and 1 before visualization. Hope that clarifies things.
Answers.UserDisplayName: Ingmar Ludwig
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T18:00:34.500Z
Answers.Text: "But sometimes these values are slightly over 1 or under 0 (at the last layer without activation function)"...doesn't it apply to specific neural network model? I thought the last layer of most CNN implement models is a dense layer. Do you mean the layer before activation in the middle?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-21T14:43:26.407Z
Answers.Text: No, I mean the last layer, which can also be a Convolutional Layer, depending on the architecture. But also a dense layer can output values that are not in the range between 0 and 1.
Answers.UserDisplayName: Ingmar Ludwig
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-29T15:19:06.523Z
Answers.LastActivityDate: 2023-09-29T15:19:06.523Z
Answers.UserDisplayName: Ingmar Ludwig
Answers.Body: <p>Great that a solution was found (clamp larger-than-one pixels' brightness before showing the image). But I suggest that you either add a sigmoid activation, or clamp the network's output directly from zero to one. This way the plotting code doesn't need to handle out-of-spec values, and in theory the model can achieve a bit smaller loss. It doesn't need try to predict exactly 1.0 or 0.0, but values like 2.3 and -0.5 are ok since they get clipped to the correct values (assuming that clipping, not sigmoid is used).</p>
  <p>Sigmoid has a bit nicer gradients than hard clipping, if the model overshoots predictions and the target value is for example 0.9 or 0.1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T13:26:38.970Z
Answers.LastActivityDate: 2023-10-30T13:26:38.970Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I have been trying to train a CNN for the super-resolution task based on the work of Dong et al., 2015 [1]. The network structure built in PyTorch is as follows:</p>
  
  <pre><code>  (0): Conv2d(1, 64, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))
    (1): ReLU()
    (2): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
    (3): ReLU()
    (4): Conv2d(32, 1, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  </code></pre>
  
  <p>I have a training dataset which consists of approximately 22.000 sub-images generated from 91 images and training is performed only on the <code>Y channel</code> of the images in <code>YCbCr color space</code>. During the training process, I used RMSE loss and calculated the PSNR (Peak Signal to Noise Ratio) from that loss. I observed that PSNR value is increasing as a result of decreasing loss as expected and as depicted in the figure. </p>
  
  <p><a href="https://i.sstatic.net/9cAKF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9cAKF.png" alt="psnr_vs_epoch_and_loss_vs_epoch"></a></p>
  
  <p>I trained the network for 25 epochs. After 10th epoch, the network is converged and PSNR value started to increase slowly. After this point, I was expecting to get even better visual outputs with higher PSNR values achieved. However, when I analyze the results of the network, there are some black pixels appearing in white spots in the output images that the network produced.</p>
  
  <p><a href="https://i.sstatic.net/6rBxm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6rBxm.png" alt="10th_epoch_vs_25th_epoch"></a></p>
  
  <p>After 25-epoch training was completed, I compared the outcome of 25th epoch (right) with that of 10th epoch (left) as you can see in the figure above.</p>
  
  <p>What might be the possible reasons for the undesired black pixels and the possible precautions that can be embedded into the network to get rid of these?</p>
  
  <p>If you would like to check my code, you can visit <a href="https://github.com/shdw2x/srcnn" rel="nofollow noreferrer">here</a>.</p>
  
  <p>[1] Dong, Chao, Chen Change Loy, Kaiming He, and Xiaoou Tang. <a href="https://arxiv.org/abs/1501.00092" rel="nofollow noreferrer">"Image Super-Resolution Using Deep Convolutional Networks."</a> IEEE Transactions on Pattern Analysis and Machine Intelligence 38, no. 2 (2015): 295-307. doi:10.1109/tpami.2015.2439281.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T15:56:35.800Z
Comments.Text: I met the similar problem like you proposed. Do you solve it later? Look forward to your help. Thanks.
Comments.UserDisplayName: 吴沛阳
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-25T11:44:20.880Z
Comments.Text: @吴沛阳 unfortunately, still don't know how to solve it.
Comments.UserDisplayName: Utku
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-29T16:14:21.203Z
Comments.Text: How about white dot?
Comments.UserDisplayName: Cloud Cho
Tags: <convolutional-neural-networks><pytorch>
Title: Super Resolution CNN generates black dots on output images
UserDisplayName: Utku
ThreadText: # Super Resolution CNN generates black dots on output images
  
  (Asked by: username_0 on 2019-08-10)
  
  I have been trying to train a CNN for the super-resolution task based on the work of Dong et al., 2015 [1]. The network structure built in PyTorch is as follows:
  
  ```
    (0): Conv2d(1, 64, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))
    (1): ReLU()
    (2): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
    (3): ReLU()
    (4): Conv2d(32, 1, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  
  ```
  
  I have a training dataset which consists of approximately 22.000 sub-images generated from 91 images and training is performed only on the `Y channel` of the images in `YCbCr color space`. During the training process, I used RMSE loss and calculated the PSNR (Peak Signal to Noise Ratio) from that loss. I observed that PSNR value is increasing as a result of decreasing loss as expected and as depicted in the figure.
  
  [![psnr_vs_epoch_and_loss_vs_epoch](https://i.sstatic.net/9cAKF.png)](https://i.sstatic.net/9cAKF.png)
  
  I trained the network for 25 epochs. After 10th epoch, the network is converged and PSNR value started to increase slowly. After this point, I was expecting to get even better visual outputs with higher PSNR values achieved. However, when I analyze the results of the network, there are some black pixels appearing in white spots in the output images that the network produced.
  
  [![10th_epoch_vs_25th_epoch](https://i.sstatic.net/6rBxm.png)](https://i.sstatic.net/6rBxm.png)
  
  After 25-epoch training was completed, I compared the outcome of 25th epoch (right) with that of 10th epoch (left) as you can see in the figure above.
  
  What might be the possible reasons for the undesired black pixels and the possible precautions that can be embedded into the network to get rid of these?
  
  If you would like to check my code, you can visit [here](https://github.com/shdw2x/srcnn).
  
  [1] Dong, Chao, Chen Change Loy, Kaiming He, and Xiaoou Tang. ["Image Super-Resolution Using Deep Convolutional Networks."](https://arxiv.org/abs/1501.00092) IEEE Transactions on Pattern Analysis and Machine Intelligence 38, no. 2 (2015): 295-307. doi:10.1109/tpami.2015.2439281.
  
  **Tags:** convolutional-neural-networks, pytorch
  
  > **Comments:**
  > **username_1**: I met the similar problem like you proposed. Do you solve it later? Look forward to your help. Thanks.  
  > **username_0**: @username_1 unfortunately, still don't know how to solve it.  
  > **username_2**: How about white dot?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-09-29. Score: 2
  
  This is a while ago, but still this problem might occur to someone.
  I encountered the same problem and found that the reason was how the resulting tensors are transformed to images. It seams that pixels with value 1.0 are shown as white while pixels with values bigger than 1.0 are interpreted as being black sometimes.
  So adding (in PyTorch)
  
  ```
  indices = tensor > 1.0
  tensor[indices] = 1.0 
  
  ```
  
  before converting to the image in the visualization method fixed the problem for me.
  
  > **Comments:**
  > **username_2**: Do you remember which layer in neural network? Was it some tensor in the last layer? Also which Super Resolution method (list at https://github.com/LoSealL/VideoSuperResolution#network-list-and-reference-updating)?  
  > **username_3**: It was not in a layer at all in that sense. The Network outputs values that are mostly between 1 and 0. But sometimes these values are slightly over 1 or under 0 if no activation function is used after the last layer (e. g. no sigmoid) . So the network gives a value whiter than white so to say. Independent of the super resolution method this might always happen without activation function. However, some visualization packets show values bigger than 1 as black. So you might fix that by clipping the values between 0 and 1 before visualization. Hope that clarifies things.  
  > **username_2**: "But sometimes these values are slightly over 1 or under 0 (at the last layer without activation function)"...doesn't it apply to specific neural network model? I thought the last layer of most CNN implement models is a dense layer. Do you mean the layer before activation in the middle?  
  > **username_3**: No, I mean the last layer, which can also be a Convolutional Layer, depending on the architecture. But also a dense layer can output values that are not in the range between 0 and 1.  
  
  ---
  
  ### Answer by username_4 on 2023-10-30. Score: 1
  
  Great that a solution was found (clamp larger-than-one pixels' brightness before showing the image). But I suggest that you either add a sigmoid activation, or clamp the network's output directly from zero to one. This way the plotting code doesn't need to handle out-of-spec values, and in theory the model can achieve a bit smaller loss. It doesn't need try to predict exactly 1.0 or 0.0, but values like 2.3 and -0.5 are ok since they get clipped to the correct values (assuming that clipping, not sigmoid is used).
  
  Sigmoid has a bit nicer gradients than hard clipping, if the model overshoots predictions and the target value is for example 0.9 or 0.1.
  
  
  ---
  
  