Answers.Body: <p>I don't think there's a way of doing what you want, at least, I've never seen such a thing (and, currently, I am not seeing how this could be done in the general case). </p>
  
  <p>The same neural network model but with different (or same) weights could have been trained with the same loss function or not. For example, although it may not be a good idea, you can train a neural network for classification with the mean squared error, as opposed to the typical cross-entropy. Moreover, even if you know the loss function that the neural network is trained with, the training data alone may not lead to the same set of weights because the actual weights depend on different (possibly stochastic) factors, such as if (or how) you shuffle the data or the batch size.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-30T13:35:00.717Z
Answers.Text: I deleted my comment as soon as I read it / it was poorly expressed and you are too fast. I realize that I neglected the architecture of the network as well
Answers.UserDisplayName: aquagremlin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-30T13:39:26.207Z
Answers.Text: The idea that led me to this question is the fact that any function can be represented as a polynomial approximation (Taylor series for example). Since the loss function is thus represented, then maybe the tools of linear algebra can be used to derive it from the weights of a model and the training data. Having such a loss function should then allow training the original network from the training data from a single densely connected layer.
Answers.UserDisplayName: aquagremlin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-30T13:42:51.620Z
Answers.Text: @aquagremlin I see your point. I am not excluding the possibility to do something in that direction, but, right now, I am not seeing a solution.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-30T13:45:45.230Z
Answers.Text: Why bother to make a loss function like this? Because I want to see if it converges faster to a smaller error. And although derived from a specific set of weights and training data, it might be applicable to similar desired models -for example instead of having a single complicated network to recognize faces, we might have several simpler single layer networks for different parts of the world. These models would smaller, faster and deployable and smaller cheaper hardware
Answers.UserDisplayName: aquagremlin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-30T13:50:21.267Z
Answers.Text: @aquagremlin When I was writing this answer, I found some papers that introduce a way of learning a loss function. For example, see the paper [Learning to Teach with Dynamic Loss Function](https://papers.nips.cc/paper/7882-learning-to-teach-with-dynamic-loss-functions.pdf).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-29T21:25:24.597Z
Answers.LastActivityDate: 2020-05-29T21:31:10.373Z
Answers.LastEditDate: 2020-05-29T21:31:10.373Z
Answers.UserDisplayName: nbro
Body: <p>There is some sort of art to using the right loss function. However, I was wondering if there is a way to derive the loss function if I gave you a neural network model (the weights) as well as the training data. </p>
  
  <p>The point of this exercise is to see what family of loss functions we would get. And how that compares to the loss function that actually gave rise to the model.</p>
  
Tags: <neural-networks><machine-learning><training><objective-functions>
Title: Is there a way of deriving a loss function given the neural network and training data?
UserDisplayName: aquagremlin
ThreadText: # Is there a way of deriving a loss function given the neural network and training data?
  
  (Asked by: username_0 on 2020-05-28)
  
  There is some sort of art to using the right loss function. However, I was wondering if there is a way to derive the loss function if I gave you a neural network model (the weights) as well as the training data.
  
  The point of this exercise is to see what family of loss functions we would get. And how that compares to the loss function that actually gave rise to the model.
  
  **Tags:** neural-networks, machine-learning, training, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-29. Score: 1 | Accepted answer
  
  I don't think there's a way of doing what you want, at least, I've never seen such a thing (and, currently, I am not seeing how this could be done in the general case).
  
  The same neural network model but with different (or same) weights could have been trained with the same loss function or not. For example, although it may not be a good idea, you can train a neural network for classification with the mean squared error, as opposed to the typical cross-entropy. Moreover, even if you know the loss function that the neural network is trained with, the training data alone may not lead to the same set of weights because the actual weights depend on different (possibly stochastic) factors, such as if (or how) you shuffle the data or the batch size.
  
  > **Comments:**
  > **username_0**: I deleted my comment as soon as I read it / it was poorly expressed and you are too fast. I realize that I neglected the architecture of the network as well  
  > **username_0**: The idea that led me to this question is the fact that any function can be represented as a polynomial approximation (Taylor series for example). Since the loss function is thus represented, then maybe the tools of linear algebra can be used to derive it from the weights of a model and the training data. Having such a loss function should then allow training the original network from the training data from a single densely connected layer.  
  > **username_1**: @aquagremlin I see your point. I am not excluding the possibility to do something in that direction, but, right now, I am not seeing a solution.  
  > **username_0**: Why bother to make a loss function like this? Because I want to see if it converges faster to a smaller error. And although derived from a specific set of weights and training data, it might be applicable to similar desired models -for example instead of having a single complicated network to recognize faces, we might have several simpler single layer networks for different parts of the world. These models would smaller, faster and deployable and smaller cheaper hardware  
  > **username_1**: @aquagremlin When I was writing this answer, I found some papers that introduce a way of learning a loss function. For example, see the paper [Learning to Teach with Dynamic Loss Function](https://papers.nips.cc/paper/7882-learning-to-teach-with-dynamic-loss-functions.pdf).  
  
  ---
  
  