Answers.Body: <p>Assuming that you're using convolutional layers, those artifacts may be related to the boundary conditions used.  The convolution kernels have a spatial support of say 3x3 pixels, meaning that the response at a position is a function of the corresponding inputs in a 3x3 neighborhood at that position.  If the position is adjacent to a boundary, say at the upper left pixel, then the south-west, west, north-west, north, and north-east pixels are outside the image region.</p>
  <p>If you only use so-called &quot;valid&quot; pixels, that is, you only get responses for those positions where the entire convolution kernel is inside the image region, then you're ignoring some information at the &quot;invalid&quot; pixels.  On the other hand, we are missing information at those invalid pixels, so an assumption must be made about those missing pixels.</p>
  <p>For your application, in the internal layers, where the expected features are gradient-like / oscillating and localized, you may wish to try zero-padding, because that's like an assumption that one doesn't assume new feature information outside the region.</p>
  <p>However, at the first layer itself, the convolution is with a natural image (as in your example), so you need to predict the pixel values at the border, so there you might want to experiment with &quot;repeat&quot; or &quot;reflecting&quot;/&quot;mirror&quot; boundary conditions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T14:56:44.780Z
Answers.Text: Thank you for your response. I'm using "same" padding in all layers, with zero padding. I'll try out the other padding modes in the initial layer, as suggested.
Answers.UserDisplayName: Susmit Agrawal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T12:33:02.047Z
Answers.Text: Using "reflect" padding on the image worked great! Thanks!
Answers.UserDisplayName: Susmit Agrawal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T14:49:52.250Z
Answers.LastActivityDate: 2022-03-22T14:49:52.250Z
Answers.UserDisplayName: Dr. Jonas August
Answers.Body: <p>That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these kind of issues.</p>
  <p>The use of L2 regularization is also a potential culprit. Pixelwise losses are also known to lead to noisy images, since they don't allow reconstruction freedom to the model.</p>
  <p>To understand why you can check this <a href="https://distill.pub/2016/deconv-checkerboard/" rel="nofollow noreferrer">blog post</a>, in this case gifs are more understandable than a thousand words, but long story short, is mostly related to emphasized overlapping region generated by the combinations of kernel/stride sizes.</p>
  <p>Solutions:</p>
  <ul>
  <li>use some more complex but not pixelwise regularization like <a href="https://deepai.org/machine-learning-glossary-and-terms/perceptual-loss-function" rel="nofollow noreferrer">perceptual loss</a></li>
  <li>remove ConvTranspose2D layers if you're using any, and move to other upsampling layers like Upsample (with bilinear or cubic interpolation)</li>
  <li>in my experience a nice trick that works pretty well in production when the artifact are only on the edges or corners is to pad the input image (with reflect setting) and then crop back the generated image in the original size. This way the model generate artifacts only in non important areas of the image.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T15:00:31.633Z
Answers.Text: I'm already using perceptual loss alongside MSE loss. Furthermore, since there is no downsampling done, there is no need for upsampling. I'll try out the input padding trick, thanks.
Answers.UserDisplayName: Susmit Agrawal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T15:04:32.320Z
Answers.Text: then I would simply remove the MSE loss, or replace it with L1 loss if the perceptual does not suffice. And if you're using an autoencoder I have a hard time believing there's no upsampling happening, how do you go from the latent bottleneck vector size to the original one otherwise?
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T15:12:12.207Z
Answers.Text: Please read the question to get the exact setup details. I'll try using L1 loss instead of L2 as suggested.
Answers.UserDisplayName: Susmit Agrawal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T15:43:10.387Z
Answers.Text: ok, if you can provide more details about the architecture maybe we can also give more detailed suggestions on that.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T14:57:39.473Z
Answers.LastActivityDate: 2022-03-22T14:57:39.473Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm training an autoencoder, that does not downsample images but processes them in the same size. For example, a 256x256 input will always be processed at 256x256 resolution, only the channels increase deeper in the network. This design is due to the next stage for which the model will be used. L2 regularization on activations is added since this is essentially a sparse autoencoder.</p>
  <p>However, I get some strange artifacts in the corners when the model converges:</p>
  <p><a href="https://i.sstatic.net/G4RXw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G4RXw.png" alt="Reference" /></a>
  <a href="https://i.sstatic.net/5TQzR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5TQzR.png" alt="Prediction" /></a></p>
  <p>The left image is the input, the image on the right is the output. The artifacts can be seen on the top-left and top-right corners of the output image.</p>
  <p>Can someone explain what causes these artifacts, and how they can be fixed?</p>
  
Tags: <convolutional-neural-networks><autoencoders>
Title: Strange artifacts in autoencoder outputs
UserDisplayName: Susmit Agrawal
ThreadText: # Strange artifacts in autoencoder outputs
  
  (Asked by: username_0 on 2022-03-22)
  
  I'm training an autoencoder, that does not downsample images but processes them in the same size. For example, a 256x256 input will always be processed at 256x256 resolution, only the channels increase deeper in the network. This design is due to the next stage for which the model will be used. L2 regularization on activations is added since this is essentially a sparse autoencoder.
  
  However, I get some strange artifacts in the corners when the model converges:
  
  [![Reference](https://i.sstatic.net/G4RXw.png)](https://i.sstatic.net/G4RXw.png)
  [![Prediction](https://i.sstatic.net/5TQzR.png)](https://i.sstatic.net/5TQzR.png)
  
  The left image is the input, the image on the right is the output. The artifacts can be seen on the top-left and top-right corners of the output image.
  
  Can someone explain what causes these artifacts, and how they can be fixed?
  
  **Tags:** convolutional-neural-networks, autoencoders
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-03-22. Score: 2 | Accepted answer
  
  Assuming that you're using convolutional layers, those artifacts may be related to the boundary conditions used. The convolution kernels have a spatial support of say 3x3 pixels, meaning that the response at a position is a function of the corresponding inputs in a 3x3 neighborhood at that position. If the position is adjacent to a boundary, say at the upper left pixel, then the south-west, west, north-west, north, and north-east pixels are outside the image region.
  
  If you only use so-called "valid" pixels, that is, you only get responses for those positions where the entire convolution kernel is inside the image region, then you're ignoring some information at the "invalid" pixels. On the other hand, we are missing information at those invalid pixels, so an assumption must be made about those missing pixels.
  
  For your application, in the internal layers, where the expected features are gradient-like / oscillating and localized, you may wish to try zero-padding, because that's like an assumption that one doesn't assume new feature information outside the region.
  
  However, at the first layer itself, the convolution is with a natural image (as in your example), so you need to predict the pixel values at the border, so there you might want to experiment with "repeat" or "reflecting"/"mirror" boundary conditions.
  
  > **Comments:**
  > **username_0**: Thank you for your response. I'm using "same" padding in all layers, with zero padding. I'll try out the other padding modes in the initial layer, as suggested.  
  > **username_0**: Using "reflect" padding on the image worked great! Thanks!  
  
  ---
  
  ### Answer by username_2 on 2022-03-22. Score: 4
  
  That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these kind of issues.
  
  The use of L2 regularization is also a potential culprit. Pixelwise losses are also known to lead to noisy images, since they don't allow reconstruction freedom to the model.
  
  To understand why you can check this [blog post](https://distill.pub/2016/deconv-checkerboard/), in this case gifs are more understandable than a thousand words, but long story short, is mostly related to emphasized overlapping region generated by the combinations of kernel/stride sizes.
  
  Solutions:
  
  * use some more complex but not pixelwise regularization like [perceptual loss](https://deepai.org/machine-learning-glossary-and-terms/perceptual-loss-function)
  * remove ConvTranspose2D layers if you're using any, and move to other upsampling layers like Upsample (with bilinear or cubic interpolation)
  * in my experience a nice trick that works pretty well in production when the artifact are only on the edges or corners is to pad the input image (with reflect setting) and then crop back the generated image in the original size. This way the model generate artifacts only in non important areas of the image.
  
  > **Comments:**
  > **username_0**: I'm already using perceptual loss alongside MSE loss. Furthermore, since there is no downsampling done, there is no need for upsampling. I'll try out the input padding trick, thanks.  
  > **username_2**: then I would simply remove the MSE loss, or replace it with L1 loss if the perceptual does not suffice. And if you're using an autoencoder I have a hard time believing there's no upsampling happening, how do you go from the latent bottleneck vector size to the original one otherwise?  
  > **username_0**: Please read the question to get the exact setup details. I'll try using L1 loss instead of L2 as suggested.  
  > **username_2**: ok, if you can provide more details about the architecture maybe we can also give more detailed suggestions on that.  
  
  ---
  
  