Body: <p>I am performing a regression task on sparse images. The images are a result of a physical process with meaningful parameters (actually, they are a superposition of cone-like shapes), and I am trying to train a regressor for these parameters.</p>
  
  <p>Here, sparse images mean that the data, and thus the expected output, is made of 2D square tensors, with only one channel, and that it is expectd that roughly 90% of the image is equal to zero. However, in my system, the data is represented as dense tensors.</p>
  
  <p>I built a neural network with an encoder mapping the image on an output for which I have chosen activation and shape such that it corresponds with those meaningful parameters.</p>
  
  <p>I then use custom layers to build an image from these parameters in a way that matches closely the physical process, and train the network by using the L2 distance between the input image and the output image.</p>
  
  <p>However, for a large set of parameters, the output image will be equal to zero, since these are sparse images. This is the case in general for the initial network.</p>
  
  <p>Is it possible that, through training, the neural network will learn its way out of this all-zero parameterization ?</p>
  
  <p>My intuition is that, in the beginning, the loss will be equal to the L2 norm of the input image, and the gradient will be uniformly zero, hence, no learning.</p>
  
  <p>Can anyone confirm ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T12:44:55.970Z
Comments.Text: Hi Alex and welcome to AI SE! Can you clarify what you mean by "sparse images"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T12:58:42.213Z
Comments.Text: Just edited the question. Thank you by the way :-)
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T16:59:23.690Z
Comments.Text: So, are you using some kind of autoencoder?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T17:31:26.817Z
Comments.Text: Yes, except for the fact that the decoder is deterministic.
Comments.UserDisplayName: Alex
Tags: <neural-networks><image-processing><regression>
Title: Can a neural network whose output is uniformly equal to zero learn its way out of it?
UserDisplayName: Alex
ThreadText: # Can a neural network whose output is uniformly equal to zero learn its way out of it?
  
  (Asked by: username_0 on 2020-04-14)
  
  I am performing a regression task on sparse images. The images are a result of a physical process with meaningful parameters (actually, they are a superposition of cone-like shapes), and I am trying to train a regressor for these parameters.
  
  Here, sparse images mean that the data, and thus the expected output, is made of 2D square tensors, with only one channel, and that it is expectd that roughly 90% of the image is equal to zero. However, in my system, the data is represented as dense tensors.
  
  I built a neural network with an encoder mapping the image on an output for which I have chosen activation and shape such that it corresponds with those meaningful parameters.
  
  I then use custom layers to build an image from these parameters in a way that matches closely the physical process, and train the network by using the L2 distance between the input image and the output image.
  
  However, for a large set of parameters, the output image will be equal to zero, since these are sparse images. This is the case in general for the initial network.
  
  Is it possible that, through training, the neural network will learn its way out of this all-zero parameterization ?
  
  My intuition is that, in the beginning, the loss will be equal to the L2 norm of the input image, and the gradient will be uniformly zero, hence, no learning.
  
  Can anyone confirm ?
  
  **Tags:** neural-networks, image-processing, regression
  > **Comments:**
  > **username_1**: Hi Alex and welcome to AI SE! Can you clarify what you mean by "sparse images"?  
  > **username_0**: Just edited the question. Thank you by the way :-)  
  > **username_1**: So, are you using some kind of autoencoder?  
  > **username_0**: Yes, except for the fact that the decoder is deterministic.  
  
  ---
  
  ## 0 answers
  
  