Answers.Body: <p>I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.</p>
  <p>The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.</p>
  <p>You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">https://www.deeplearningbook.org/</a></p>
  <p>or here : <a href="https://en.wikipedia.org/wiki/Loss_functions_for_classification" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Loss_functions_for_classification</a></p>
  <p>I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in <em>A Practical Bayesian Framework for Backpropagation networks</em> by McKay 1992.</p>
  <p>It is explained very well in a video series by Hinton on youtube here:
  <a href="https://www.youtube.com/watch?v=YcwZFNd3UvI" rel="nofollow noreferrer">https://www.youtube.com/watch?v=YcwZFNd3UvI</a></p>
  <p>Hopefully these references help you I wish I could explain in greater detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-05T03:18:41.143Z
Answers.LastActivityDate: 2021-02-05T03:18:41.143Z
Answers.UserDisplayName: NicoFish
Body: <p>I am reading article <a href="https://allenai.org/paper-appendix/emnlp2017-wt/" rel="nofollow noreferrer">https://allenai.org/paper-appendix/emnlp2017-wt/</a> <a href="http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf" rel="nofollow noreferrer">http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf</a> about training neural network and the loss function is mentioned on page 6 chapter 3.4 - this loss function <code>O(theta)</code> is expressed as marginal loglikelihood objective function. I simply does not understand this. The neural network generates logical expression (query) from some question in natural language. The network is trained using question-answer pairs. One could expect that simple sum of correct-1/incorrect=0 result could be good loss function. But there is strange expression that involves <code>P(l|qi, Ti; theta)</code> that is not mentioned in the article. What is meant by this <code>P</code> function? As I understand, then many logical forms <code>l</code> are generated externally for some question <code>qi</code>. But further I can not understand this. The mentioned article largely builds on other article <a href="http://www.aclweb.org/anthology/P16-1003" rel="nofollow noreferrer">http://www.aclweb.org/anthology/P16-1003</a> from which it borrows some terms and ideas.</p>
  
  <p>It is said that <code>l</code> is treated as latent variable and <code>P</code> seems to be some kind of probability. Of course, we should assign the greated probability to the right logical form <code>l</code>, but where can I find this assignment. Does training/supervision data should contain this probability function for training/supervision data?</p>
  
Tags: <neural-networks><natural-language-processing><long-short-term-memory><objective-functions>
Title: How to understand marginal loglikelihood objective function as loss function (explanation of an article)?
UserDisplayName: TomR
ThreadText: # How to understand marginal loglikelihood objective function as loss function (explanation of an article)?
  
  (Asked by: username_0 on 2018-10-17)
  
  I am reading article <https://allenai.org/paper-appendix/emnlp2017-wt/> <http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf> about training neural network and the loss function is mentioned on page 6 chapter 3.4 - this loss function `O(theta)` is expressed as marginal loglikelihood objective function. I simply does not understand this. The neural network generates logical expression (query) from some question in natural language. The network is trained using question-answer pairs. One could expect that simple sum of correct-1/incorrect=0 result could be good loss function. But there is strange expression that involves `P(l|qi, Ti; theta)` that is not mentioned in the article. What is meant by this `P` function? As I understand, then many logical forms `l` are generated externally for some question `qi`. But further I can not understand this. The mentioned article largely builds on other article <http://www.aclweb.org/anthology/P16-1003> from which it borrows some terms and ideas.
  
  It is said that `l` is treated as latent variable and `P` seems to be some kind of probability. Of course, we should assign the greated probability to the right logical form `l`, but where can I find this assignment. Does training/supervision data should contain this probability function for training/supervision data?
  
  **Tags:** neural-networks, natural-language-processing, long-short-term-memory, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-05. Score: 1
  
  I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.
  
  The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.
  
  You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: <https://www.deeplearningbook.org/>
  
  or here : <https://en.wikipedia.org/wiki/Loss_functions_for_classification>
  
  I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in *A Practical Bayesian Framework for Backpropagation networks* by McKay 1992.
  
  It is explained very well in a video series by Hinton on youtube here:
  <https://www.youtube.com/watch?v=YcwZFNd3UvI>
  
  Hopefully these references help you I wish I could explain in greater detail.
  
  
  ---
  
  