Body: <p>I want to train a neural network on some input data from a probability distribution (say a Gaussian). The loss function would normally be <span class="math-container">$-\sum\log(f(x_i))$</span>, where the sum is over the whole data (or in this case a mini batch) and <span class="math-container">$f$</span> is the NN function. However I need to enforce the fact that <span class="math-container">$\int_0^\infty f(x)dx=1$</span>, in order for <span class="math-container">$f$</span> to be a real probability distribution. How can I add that to the loss function? Thank you! </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-08T05:00:14.227Z
Comments.Text: I am given data points generated from a function (distribution) as input, and I want to train a NN to approximate that function. I am not sure if it is classification or regression. I just want to train the network to approximate f.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-08T05:54:14.930Z
Comments.Text: Both the input and the output are one single node, yes. It can be shown mathematically that given data generated by a probability distribution, the function that generated the data is the one minimizing the loss function I wrote there. I just need to impose that normalization (otherwise the constant function equal to 1 would give the smallest loss). But I am not sure how to do that.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-08T19:15:47.210Z
Comments.Text: The inputs are generated randomly from the distribution. They are not continuous, unless you generate infinitely many, which is not possible.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-29T15:31:38.740Z
Comments.Text: @AlexMarshall Your description of the function **$f$** is confusing. *...f  is the NN function....* and then, *...f  to be a real probability distribution...*. And, previously you desribe the loss: *The loss function would normally be −∑log(f(xi)),...*. I think you need to clear up the question and method a bit more.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T06:28:51.897Z
Comments.Text: @naive So I have an unknown probability distribution, let's say it's a gaussian distribution (this is also a function, as I can write it as f(x, mu, sigma)). I have some samples from the distribution and I want to build a NN such that given these sampled numbers as input, the NN learns to approximate the real (unknown) distribution. So if I pass a new number to the NN, it will give me something close to the value of the gaussian at that number.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T06:31:23.400Z
Comments.Text: @naive It can be shown mathematically that the loss function that I mentioned there is minimized only when the NN approximates the real function (intuitively, you want to find the function/distribution for which the numbers that you sampled are the most likely to appear). However, a probability distribution (such as a gaussian) must be normalized. My question is how do I impose the normalization condition to the NN such that it won't give me, for example numbers bigger than one, or numbers that would add up to more than one.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T07:04:42.197Z
Comments.Text: I think your first comment is clear. But, the second conflates the idea of a probability distribution function ( pdf) and probability.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T07:08:43.760Z
Comments.Text: From what I understood, your problem is of approximating a probability distribution function. This is not an easy task. You might find different probability distribution functions with similar losses for the training set that you provide. And, for the matter of normal distribution -- the normalizing constant can be used which is a function of the variance -- that I assume your function would have to approximate.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T07:30:15.857Z
Comments.Text: @naive oh sorry, you are right, I mean pdf. But yes, there could be many pdf that could produce the samples that I have with a high probability. But that's ok. I don't need the exact pdf, just something approximate. But that is mainly a second worry. the first one is how can I define a loss function itself that will force my NN the learn a good enough pdf? And in the case of the gaussian, yes, the NN should discover on its own that normalization constant. I am just not sure how to approach the problem.
Comments.UserDisplayName: Alex Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T10:14:28.403Z
Comments.Text: For defining a loss function you need to have the actual output and predicted output. According to your description there does not seem to have an actual output. It seems this problem can be handled better through unsupervised learning techniques.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-01T16:18:09.373Z
Comments.Text: @naive But what is wrong with the loss I suggested? Shouldn't the network get pretty close to the tru pdf by using that loss?
Comments.UserDisplayName: Alex Marshall
Tags: <objective-functions><probability-distribution>
Title: Unit integral condition on the output layer
UserDisplayName: Alex Marshall
ThreadText: # Unit integral condition on the output layer
  
  (Asked by: username_0 on 2019-06-08)
  
  I want to train a neural network on some input data from a probability distribution (say a Gaussian). The loss function would normally be $-\sum\log(f(x\_i))$, where the sum is over the whole data (or in this case a mini batch) and $f$ is the NN function. However I need to enforce the fact that $\int\_0^\infty f(x)dx=1$, in order for $f$ to be a real probability distribution. How can I add that to the loss function? Thank you!
  
  **Tags:** objective-functions, probability-distribution
  
  > **Comments:**
  > **username_0**: I am given data points generated from a function (distribution) as input, and I want to train a NN to approximate that function. I am not sure if it is classification or regression. I just want to train the network to approximate f.  
  > **username_0**: Both the input and the output are one single node, yes. It can be shown mathematically that given data generated by a probability distribution, the function that generated the data is the one minimizing the loss function I wrote there. I just need to impose that normalization (otherwise the constant function equal to 1 would give the smallest loss). But I am not sure how to do that.  
  > **username_0**: The inputs are generated randomly from the distribution. They are not continuous, unless you generate infinitely many, which is not possible.  
  > **username_1**: @username_0 Your description of the function **$f$** is confusing. *...f  is the NN function....* and then, *...f  to be a real probability distribution...*. And, previously you desribe the loss: *The loss function would normally be −∑log(f(xi)),...*. I think you need to clear up the question and method a bit more.  
  > **username_0**: @username_1 So I have an unknown probability distribution, let's say it's a gaussian distribution (this is also a function, as I can write it as f(x, mu, sigma)). I have some samples from the distribution and I want to build a NN such that given these sampled numbers as input, the NN learns to approximate the real (unknown) distribution. So if I pass a new number to the NN, it will give me something close to the value of the gaussian at that number.  
  > **username_0**: @username_1 It can be shown mathematically that the loss function that I mentioned there is minimized only when the NN approximates the real function (intuitively, you want to find the function/distribution for which the numbers that you sampled are the most likely to appear). However, a probability distribution (such as a gaussian) must be normalized. My question is how do I impose the normalization condition to the NN such that it won't give me, for example numbers bigger than one, or numbers that would add up to more than one.  
  > **username_1**: I think your first comment is clear. But, the second conflates the idea of a probability distribution function ( pdf) and probability.  
  > **username_1**: From what I understood, your problem is of approximating a probability distribution function. This is not an easy task. You might find different probability distribution functions with similar losses for the training set that you provide. And, for the matter of normal distribution -- the normalizing constant can be used which is a function of the variance -- that I assume your function would have to approximate.  
  > **username_0**: @username_1 oh sorry, you are right, I mean pdf. But yes, there could be many pdf that could produce the samples that I have with a high probability. But that's ok. I don't need the exact pdf, just something approximate. But that is mainly a second worry. the first one is how can I define a loss function itself that will force my NN the learn a good enough pdf? And in the case of the gaussian, yes, the NN should discover on its own that normalization constant. I am just not sure how to approach the problem.  
  > **username_1**: For defining a loss function you need to have the actual output and predicted output. According to your description there does not seem to have an actual output. It seems this problem can be handled better through unsupervised learning techniques.  
  > **username_0**: @username_1 But what is wrong with the loss I suggested? Shouldn't the network get pretty close to the tru pdf by using that loss?  
  
  ---
  
  ## 0 answers
  
  