Body: <p>I am looking at a problem which can be distilled as follows:  I have a phenomenon which can be modeled as a probability density function which is "messy" in that it sums to unity over its support but is somewhat jagged and spiky, and does not correspond to any particular textbook function.  It takes considerable amounts of time to generate these experimental density functions, along with conditional data for machine learning, but I have them.  I also have a crude model which runs quickly but performs poorly, i.e., generates poor quality density functions.</p>
  
  <p>I would like to train a neural network to transform the crude estimated pdfs to something closer to the experimentally generated pdfs, if possible.</p>
  
  <p>To investigate this, I've further reduced this to the most toy-like toy problem I can think of:  Feeding a narrow, smooth (relatively narrow) normal curve into a 1D convolutional neural network, and trying to transform it to a similar narrow curve with a different mean.  Both input and output have fine enough support (101 points) to be considered as a smooth pdf.</p>
  
  <p>Here is the crux of the problem I think I have: <em>I do not know what a good loss function is for this problem.</em></p>
  
  <p>L1, L2 and similar losses are useless, given that once the non-zero parts of the pdfs are non-overlapping, it doesn't matter how far apart the means are, the loss remains the same. </p>
  
  <p>I have been experimenting with <a href="https://www.kernel-operations.io/geomloss/api/pytorch-api.html" rel="nofollow noreferrer">Sinkhorn approximations</a> to optimal transport, to properly capture the intuition of "distance" but somewhat surprisingly these have not been helpful either.  I think part of the problem may be an (unavoidable?) numerical stability issue related to the support, but I would not stake hard money on that assumption.  </p>
  
  <p>(If support is at percentiles on the [0,1] it is quite instructive (and dismaying) to look at the sinkhorn loss for normal functions with the mean directly on a point of support, vs normal functions with the mean directly between two points of support.)</p>
  
  <p>For a problem in this vein, are there any recommended loss functions (preferably supported by or easily implement in PyTorch) which might work better?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T20:51:03.207Z
Comments.Text: Moderators, I waffled between asking this here (for the machine learning application) vs Cross Validated (for the statistics insight.)  If it seems better posed there, let me know and I can remove and repost.
Comments.UserDisplayName: Novak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T21:04:21.677Z
Comments.Text: Hi. What is the "crude deterministic model" used for in your context? You also say "I would like to train a neural network to transform the crude estimate to the experimental data", but to transform to what? Maybe edit your question to specify (even though from the title it seems you want to convert a "messy probability distribution" to another probability distribution).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T21:11:25.257Z
Comments.Text: @nbro I'm not sure what you're asking.  Yes, I want to feed in the crude estimated pdf (along with conditional data) and get out a substantially better estimate of the messy, experimentally generated pdf, if that's what you mean.
Comments.UserDisplayName: Novak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T21:13:56.353Z
Comments.Text: You say "I also have a crude deterministic model which runs quickly but performs poorly.". What does "crude deterministic model" mean in this context? Model of what? What is the relationship between this model and the probability density that models the phenomenon?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T21:21:46.180Z
Comments.Text: @nbro I'm trying not to get sidetracked into the domain knowledge of the problem.  'Deterministic' might be the problem-- it is in retrospect a poorly chosen term.  I mean I have a crude, computationally cheap model that outputs crude, poorly performing density functions which I would like to improve by training against expensive density functions that I consider as ground truth.
Comments.UserDisplayName: Novak
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-07T21:27:09.477Z
Comments.Text: Updated the question in a fashion I hope is more clear.
Comments.UserDisplayName: Novak
Tags: <ai-design><objective-functions><probability-distribution>
Title: Which loss functions for transforming a density function to another density function?
UserDisplayName: Novak
ThreadText: # Which loss functions for transforming a density function to another density function?
  
  (Asked by: username_0 on 2019-07-07)
  
  I am looking at a problem which can be distilled as follows: I have a phenomenon which can be modeled as a probability density function which is "messy" in that it sums to unity over its support but is somewhat jagged and spiky, and does not correspond to any particular textbook function. It takes considerable amounts of time to generate these experimental density functions, along with conditional data for machine learning, but I have them. I also have a crude model which runs quickly but performs poorly, i.e., generates poor quality density functions.
  
  I would like to train a neural network to transform the crude estimated pdfs to something closer to the experimentally generated pdfs, if possible.
  
  To investigate this, I've further reduced this to the most toy-like toy problem I can think of: Feeding a narrow, smooth (relatively narrow) normal curve into a 1D convolutional neural network, and trying to transform it to a similar narrow curve with a different mean. Both input and output have fine enough support (101 points) to be considered as a smooth pdf.
  
  Here is the crux of the problem I think I have: *I do not know what a good loss function is for this problem.*
  
  L1, L2 and similar losses are useless, given that once the non-zero parts of the pdfs are non-overlapping, it doesn't matter how far apart the means are, the loss remains the same.
  
  I have been experimenting with [Sinkhorn approximations](https://www.kernel-operations.io/geomloss/api/pytorch-api.html) to optimal transport, to properly capture the intuition of "distance" but somewhat surprisingly these have not been helpful either. I think part of the problem may be an (unavoidable?) numerical stability issue related to the support, but I would not stake hard money on that assumption.
  
  (If support is at percentiles on the [0,1] it is quite instructive (and dismaying) to look at the sinkhorn loss for normal functions with the mean directly on a point of support, vs normal functions with the mean directly between two points of support.)
  
  For a problem in this vein, are there any recommended loss functions (preferably supported by or easily implement in PyTorch) which might work better?
  
  **Tags:** ai-design, objective-functions, probability-distribution
  > **Comments:**
  > **username_0**: Moderators, I waffled between asking this here (for the machine learning application) vs Cross Validated (for the statistics insight.)  If it seems better posed there, let me know and I can remove and repost.  
  > **username_1**: Hi. What is the "crude deterministic model" used for in your context? You also say "I would like to train a neural network to transform the crude estimate to the experimental data", but to transform to what? Maybe edit your question to specify (even though from the title it seems you want to convert a "messy probability distribution" to another probability distribution).  
  > **username_0**: @nbro I'm not sure what you're asking.  Yes, I want to feed in the crude estimated pdf (along with conditional data) and get out a substantially better estimate of the messy, experimentally generated pdf, if that's what you mean.  
  > **username_1**: You say "I also have a crude deterministic model which runs quickly but performs poorly.". What does "crude deterministic model" mean in this context? Model of what? What is the relationship between this model and the probability density that models the phenomenon?  
  > **username_0**: @nbro I'm trying not to get sidetracked into the domain knowledge of the problem.  'Deterministic' might be the problem-- it is in retrospect a poorly chosen term.  I mean I have a crude, computationally cheap model that outputs crude, poorly performing density functions which I would like to improve by training against expensive density functions that I consider as ground truth.  
  > **username_0**: Updated the question in a fashion I hope is more clear.  
  
  ---
  
  ## 0 answers
  
  