Body: <p>My question may be a bit hard to explain...
  My neural network learns a categorical distribution, which serves as an index. This index will look up the value (= action_mean) in Input 2.</p>
  <p>From this action_mean, I create a normal distribution where the network has to learn to adjust the standard deviation. The output of the network is a sample of this normal distribution.</p>
  <p>Since the value of action_mean is directly taken from the input, somehow the gradient can't be computed or gives <em>Nones</em>, respectively, because the output of the net is not completely connected with the input.</p>
  <p>Would there be a way to link my action_mean with the input value, without changing the input values itself? To describe my problem, I attached a simplified computational graph how tensorboard shows it.</p>
  <p>I would be very thankful for any help!</p>
  <p><a href="https://i.sstatic.net/Q1uAk.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Q1uAk.jpg" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T01:16:02.100Z
Comments.Text: This part "_Since the value of action_mean is directly taken from the input, somehow the gradient can't be computed or gives Nones, respectively, because the output of the net is not completely connected with the input._" is not clear to me. It's not clear to me why you can't directly connect the output of one NN to the input of another NN. What is this "action mean" supposed to represent for the second network? Are you training both NNs jointly?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T11:06:02.860Z
Comments.Text: @nbro It is only one network and action_mean should represent the mean value of a normal distribution. I can write it in pseudocode for better clarity:
  
  `def forward(inp1, inp2):
  x=softmax(layer1(inp1))
  dist1 = Canonical(x)
  index = dist1.sample()
  action_mean = inp2[index]
  dist2 = Normal(action_mean, action_std)`
Comments.UserDisplayName: thsolyt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T11:13:58.850Z
Comments.Text: @nbro where action_std is a parameter defined in _init_. I dont know why but at that point where action_mean is defined, the network seems to be disconnected somehow. Overall, I need a constrained Normal distribution for certain regions and these regions will be provided by the canonical distribution and input2.
Comments.UserDisplayName: thsolyt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T13:07:52.800Z
Comments.Text: When you say "I dont know why but at that point where action_mean is defined, the network seems to be disconnected somehow. ", what do you mean by "it seems to be disconnected somehow"? Are you getting an error that tells you that you cannot back-propagate? That may be because something is not differentiable in your model (don't know, just guessing). Which library are you using? PyTorch?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-21T19:17:24.073Z
Comments.Text: @nbro yes indeed, there is an error when back-propagating. I inspected the gradients and some of them are *None*. The problem lies in the definition of action_mean, which is just a value of input2 (depending on the index) without layers or any other operation in between. So, it cant propagate the gradient back to the input. And yes, I am using PyTorch. I hope you see my problem now...
Comments.UserDisplayName: thsolyt
Tags: <neural-networks><machine-learning><reinforcement-learning><gradient-descent><graphs>
Title: How to use unmodified input in neural network?
UserDisplayName: thsolyt
ThreadText: # How to use unmodified input in neural network?
  
  (Asked by: username_0 on 2020-11-19)
  
  My question may be a bit hard to explain...
  My neural network learns a categorical distribution, which serves as an index. This index will look up the value (= action\_mean) in Input 2.
  
  From this action\_mean, I create a normal distribution where the network has to learn to adjust the standard deviation. The output of the network is a sample of this normal distribution.
  
  Since the value of action\_mean is directly taken from the input, somehow the gradient can't be computed or gives *Nones*, respectively, because the output of the net is not completely connected with the input.
  
  Would there be a way to link my action\_mean with the input value, without changing the input values itself? To describe my problem, I attached a simplified computational graph how tensorboard shows it.
  
  I would be very thankful for any help!
  
  [![enter image description here](https://i.sstatic.net/Q1uAk.jpg)](https://i.sstatic.net/Q1uAk.jpg)
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, gradient-descent, graphs
  > **Comments:**
  > **username_1**: This part "_Since the value of action_mean is directly taken from the input, somehow the gradient can't be computed or gives Nones, respectively, because the output of the net is not completely connected with the input._" is not clear to me. It's not clear to me why you can't directly connect the output of one NN to the input of another NN. What is this "action mean" supposed to represent for the second network? Are you training both NNs jointly?  
  > **username_0**: @nbro It is only one network and action_mean should represent the mean value of a normal distribution. I can write it in pseudocode for better clarity:  `def forward(inp1, inp2): x=softmax(layer1(inp1)) dist1 = Canonical(x) index = dist1.sample() action_mean = inp2[index] dist2 = Normal(action_mean, action_std)`  
  > **username_0**: @nbro where action_std is a parameter defined in _init_. I dont know why but at that point where action_mean is defined, the network seems to be disconnected somehow. Overall, I need a constrained Normal distribution for certain regions and these regions will be provided by the canonical distribution and input2.  
  > **username_1**: When you say "I dont know why but at that point where action_mean is defined, the network seems to be disconnected somehow. ", what do you mean by "it seems to be disconnected somehow"? Are you getting an error that tells you that you cannot back-propagate? That may be because something is not differentiable in your model (don't know, just guessing). Which library are you using? PyTorch?  
  > **username_0**: @nbro yes indeed, there is an error when back-propagating. I inspected the gradients and some of them are *None*. The problem lies in the definition of action_mean, which is just a value of input2 (depending on the index) without layers or any other operation in between. So, it cant propagate the gradient back to the input. And yes, I am using PyTorch. I hope you see my problem now...  
  
  ---
  
  ## 0 answers
  
  