Answers.Body: <p>I suspect you could be reinventing the wheel a little bit here, because some of the description of what this task is <em>for</em> appears to match to a Reinforcement Learning (RL) scenario. You may find if you research approaches using neural networks to solve RL problems, that you get a lot of insight into things that could help with your problem.</p>
  <p>However, there is also a general answer using the approach from RL <em>Policy Gradients</em>, assuming your NN output layer is a multiclass classifier for which action to take.</p>
  <ol>
  <li>You need to store inputs to the NN used to generate actions (let's call them <span class="math-container">$S_t$</span>), until you have the associated score function results. Let's call the score function value <span class="math-container">$G$</span>, and it will be the same for consecutive samples of <span class="math-container">$S_t$</span> - each group that ends with a known score is an &quot;episode&quot;.</li>
  <li>Along with those inputs, you need to store the <em>actual</em> action taken when that input was supplied, let's call that <span class="math-container">$A_t$</span>. This action <em>can</em> be different to the argmax choice from the classifier. You should be sampling actions according to the probabilities from the classifier (a different sampling approach is possible in general, but won't work well with the rest of the instructions here, because we rely on seeing the different scores in correct ratios).</li>
  <li>Once you have some results with score functions - ideally from several runs to reduce correlation between inputs - then you can use them to construct a minibatch for supervised learning. Retrospectively set a bunch of <span class="math-container">$G_t$</span> to match the <span class="math-container">$S_t$</span> and <span class="math-container">$A_t$</span> values.</li>
  <li>Calculate the &quot;ground truth&quot; for each input <span class="math-container">$S_t$</span> as the action that was taken <span class="math-container">$A_t$</span>. It doesn't matter whether the action was the best or correct one to take, it matters that it was the one that was actually taken, and as above, ideally sampled according to your classifier probabilities.</li>
  <li>Calculate the parameter gradient for each input, output combination as normal for your neural network for supervised learning and multiclass cross-entropy loss . . . but</li>
  <li>Before you add to the total gradient for the minibatch, multiply the weights gradient by a normalised <span class="math-container">$G_t$</span>. You can change the normalisation parameters over time - typically you subtract some average <span class="math-container">$\bar{G}$</span> seen so far (e.g. running average of results from last 100 episodes) and maybe multiply by some value to put into an easy to handle range, so <span class="math-container">$\nabla_{\theta, t} \leftarrow \beta(G_t - \bar{G})\nabla_{\theta, t}$</span>. This is the &quot;trick&quot; from policy gradients that means the neural network will learn to avoid low total scores and find high total scores on average across many updates. The offset is not required, but makes the learning process faster and more stable. The multiplication is not required either, provided you are happy to try different learning rates instead. So the absolute bare bones version is <span class="math-container">$\nabla_{\theta, t} \leftarrow G_t\nabla_{\theta, t}$</span></li>
  <li>Complete the minibatch by taking a positive gradient step on the mean gradient (perhaps multiplied by some learning rate, or modified with an optimiser like Adam)</li>
  </ol>
  <p>This is pretty much <em>REINFORCE with baseline</em>, a policy gradient method from RL, as applied to discrete action space and a softmax action classifier. It can in theory be applied in scenarios which are not technically setup as RL problems, as long as you can follow the procedure, and it reliably means something to generate a &quot;good&quot; sequence of output choices based on available inputs, by some measure (your score in this case).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T21:18:05.050Z
Answers.Text: Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.
Answers.UserDisplayName: IAQuestions
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T08:09:21.827Z
Answers.Text: @IAQuestions You should be able to vectorise the process described above. The difficult part is vectorising your environment and environmental responses, and typically that's impossible for MDPs. But if you have a way to vectorise assessing agent behaviour in your problem, all of steps 1-7 above are vectorisable. Steps 3-7 are vectorisable regardless and convert your scores to useful gradient.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T13:01:43.420Z
Answers.LastActivityDate: 2023-08-31T18:45:27.430Z
Answers.LastEditDate: 2023-08-31T18:45:27.430Z
Answers.UserDisplayName: Neil Slater
Body: <p>Loss_Function/Maximize_Function/Score_Function, CustomLoss, pytorch. Using Custom Loss for Maximizing Score in PyTorch</p>
  <p>I'm using a PyTorch model with an LSTM input layer, a linear hidden layer, and 3 neurons in the output layer with a softmax activation function.</p>
  <p>Instead of using a loss function (such as nn.MSELoss()) and passing the model's prediction and its target (loss = criterion(predictions, target)), I would like to use a custom scoring function that I've created. This scoring function is represented as score = get_score(predictions), which implies a concept similar to maximizing (predictions, score).</p>
  <p>However, since I don't possess a target for each input value of the model, the model would be evaluated based on its score (which is a single number). It appears that considering a single value as a basis for assessing a set of predictions across the entire dataset might not be valid.</p>
  <p>Is something like this possible? The times I've attempted it, I often encounter an error during the 'loss'.backward() phase, as it fails to trace the necessary gradients for backpropagation.</p>
  <p>I didn't put the code because it's a very big code, but I'll make a simple example.</p>
  <pre><code>class Model_123(nn.Module):
      def __init__(self):
          super().__init__()
          self.linear1 = nn.Linear(10, 3)
  
      def forward(self, x):
          out = self.linear1(x)
          out = F.softmax(out, dim=1)
          return out 
  
  dataset = torch.randn(10, 10)
  model = Model_123()
  
  custom_loss_score = CustomLoss_Score()  # This is what I want to figure out
  for epoch in range(epochs):
  
      model.train()
      optimizer.zero_grad()
      predictions = model(dataset)
      score = random.uniform(0, 100)
  
      # Random score just for demonstration, but it would be calculated based on the model's actions.
      # Using torch.argmax on the predictions made by the model for that input value.
      # This gives me a success rate, where correctly predicting action 0 or action 1 gets +1,
      # while action 2 means doing nothing. In a situation where the model correctly predicts only one action,
      # between actions 0 and 1, and everything else is action 2, leading to a 100% success rate.
      # This scenario is not good as there's no criteria; it just guessed one action and got 100% score.
      # It would be better if the model correctly predicted 80% of actions with a 70% operation rate,
      # which would be a better outcome. So, based on the success rate, I create another function
      # to calculate the score, a relation between the success rate and the number of operations performed.
      # This is a much larger code that I haven't included here.
  
      loss_score = custom_loss_score(predictions, score)  # Maximize the score
      loss_score.backward()
      optimizer.step()
  </code></pre>
  <p>This scoring mechanism, represented as score, would ideally be maximized by adjusting the model's parameters. However, I'm unsure about how to define the CustomLoss_Score() function that would achieve this.</p>
  <p>The score is generated based on the model's actions, and it's calculated using a relationship between the accuracy rate and the number of operations performed. The intention is to maximize this score rather than minimizing a traditional loss function.</p>
  <p>The reason I'm not using Reinforcement Learning is that in this case, I would have to loop through all states (or input data), which takes more time. On the other hand, with supervised learning, I can approach it in a vectorized manner. I'm following this strategy because I'm also utilizing training based on genetic algorithms, which involve multiple individuals (sets of model weights) making predictions on the data. If I were to use Reinforcement Learning, each individual would have to loop through the states, and this would be quite time-consuming, especially with a population size of, for example, 300 individuals.</p>
  <p>Hence, I'm employing a training approach based on gradient descent, along with training using genetic algorithms and artificial selection.</p>
  <p>When the model seems to be stuck in a particular place, I take the weights from that model and generate a population of individuals. Genetic algorithms can guide the weights to places where gradient descent might struggle, and vice versa. When one approach isn't yielding improvements, I switch to the other approach.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T07:23:09.797Z
Comments.Text: Is there a reason that you cannot score individual predictions? There might be a connection to Reinforcement Learning here, which could give you the tools to solve that.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T10:32:11.080Z
Comments.Text: As @NeilSlater pointed out, just check any PyTorch implementation of policy gradient  for example https://github.com/Finspire13/pytorch-policy-gradient-example/blob/eded52d0d4f56b2d6f72394c083dbd70029294fb/pg.py#L130-L131C32
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T14:02:16.140Z
Comments.Text: See this answer: https://ai.stackexchange.com/questions/41264/how-to-do-backpropagation-with-argmax/41312#41312; it's a similar setup.
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T21:19:39.407Z
Comments.Text: Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.
Comments.UserDisplayName: IAQuestions
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-01T23:18:04.297Z
Comments.Text: We weren't suggesting to use RL, we were saying that you can look at the code and take inspiration from it as RL can be considered a Score optimization more than a Loss since you don't have a target
Comments.UserDisplayName: Alberto
Tags: <machine-learning><deep-learning><pytorch><loss>
Title: How to transform a loss function into a score function?
UserDisplayName: IAQuestions
ThreadText: # How to transform a loss function into a score function?
  
  (Asked by: username_0 on 2023-08-30)
  
  Loss_Function/Maximize_Function/Score_Function, CustomLoss, pytorch. Using Custom Loss for Maximizing Score in PyTorch
  I'm using a PyTorch model with an LSTM input layer, a linear hidden layer, and 3 neurons in the output layer with a softmax activation function.
  Instead of using a loss function (such as nn.MSELoss()) and passing the model's prediction and its target (loss = criterion(predictions, target)), I would like to use a custom scoring function that I've created. This scoring function is represented as score = get_score(predictions), which implies a concept similar to maximizing (predictions, score).
  However, since I don't possess a target for each input value of the model, the model would be evaluated based on its score (which is a single number). It appears that considering a single value as a basis for assessing a set of predictions across the entire dataset might not be valid.
  Is something like this possible? The times I've attempted it, I often encounter an error during the 'loss'.backward() phase, as it fails to trace the necessary gradients for backpropagation.
  I didn't put the code because it's a very big code, but I'll make a simple example.
  class Model_123(nn.Module):
      def __init__(self):
          super().__init__()
          self.linear1 = nn.Linear(10, 3)
  
      def forward(self, x):
          out = self.linear1(x)
          out = F.softmax(out, dim=1)
          return out 
  
  dataset = torch.randn(10, 10)
  model = Model_123()
  
  custom_loss_score = CustomLoss_Score()  # This is what I want to figure out
  for epoch in range(epochs):
  
      model.train()
      optimizer.zero_grad()
      predictions = model(dataset)
      score = random.uniform(0, 100)
  
      # Random score just for demonstration, but it would be calculated based on the model's actions.
      # Using torch.argmax on the predictions made by the model for that input value.
      # This gives me a success rate, where correctly predicting action 0 or action 1 gets +1,
      # while action 2 means doing nothing. In a situation where the model correctly predicts only one action,
      # between actions 0 and 1, and everything else is action 2, leading to a 100% success rate.
      # This scenario is not good as there's no criteria; it just guessed one action and got 100% score.
      # It would be better if the model correctly predicted 80% of actions with a 70% operation rate,
      # which would be a better outcome. So, based on the success rate, I create another function
      # to calculate the score, a relation between the success rate and the number of operations performed.
      # This is a much larger code that I haven't included here.
  
      loss_score = custom_loss_score(predictions, score)  # Maximize the score
      loss_score.backward()
      optimizer.step()
  
  This scoring mechanism, represented as score, would ideally be maximized by adjusting the model's parameters. However, I'm unsure about how to define the CustomLoss_Score() function that would achieve this.
  The score is generated based on the model's actions, and it's calculated using a relationship between the accuracy rate and the number of operations performed. The intention is to maximize this score rather than minimizing a traditional loss function.
  The reason I'm not using Reinforcement Learning is that in this case, I would have to loop through all states (or input data), which takes more time. On the other hand, with supervised learning, I can approach it in a vectorized manner. I'm following this strategy because I'm also utilizing training based on genetic algorithms, which involve multiple individuals (sets of model weights) making predictions on the data. If I were to use Reinforcement Learning, each individual would have to loop through the states, and this would be quite time-consuming, especially with a population size of, for example, 300 individuals.
  Hence, I'm employing a training approach based on gradient descent, along with training using genetic algorithms and artificial selection.
  When the model seems to be stuck in a particular place, I take the weights from that model and generate a population of individuals. Genetic algorithms can guide the weights to places where gradient descent might struggle, and vice versa. When one approach isn't yielding improvements, I switch to the other approach.
  
  **Tags:** machine-learning, deep-learning, pytorch, loss
  > **Comments:**
  > **username_1**: Is there a reason that you cannot score individual predictions? There might be a connection to Reinforcement Learning here, which could give you the tools to solve that.  
  > **username_2**: As @NeilSlater pointed out, just check any PyTorch implementation of policy gradient  for example https://github.com/Finspire13/pytorch-policy-gradient-example/blob/eded52d0d4f56b2d6f72394c083dbd70029294fb/pg.py#L130-L131C32  
  > **username_3**: See this answer: https://ai.stackexchange.com/questions/41264/how-to-do-backpropagation-with-argmax/41312#41312; it's a similar setup.  
  > **username_0**: Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.  
  > **username_2**: We weren't suggesting to use RL, we were saying that you can look at the code and take inspiration from it as RL can be considered a Score optimization more than a Loss since you don't have a target  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-31. Score: 1
  
  I suspect you could be reinventing the wheel a little bit here, because some of the description of what this task is *for* appears to match to a Reinforcement Learning (RL) scenario. You may find if you research approaches using neural networks to solve RL problems, that you get a lot of insight into things that could help with your problem.
  
  However, there is also a general answer using the approach from RL *Policy Gradients*, assuming your NN output layer is a multiclass classifier for which action to take.
  
  1. You need to store inputs to the NN used to generate actions (let's call them $S\_t$), until you have the associated score function results. Let's call the score function value $G$, and it will be the same for consecutive samples of $S\_t$ - each group that ends with a known score is an "episode".
  2. Along with those inputs, you need to store the *actual* action taken when that input was supplied, let's call that $A\_t$. This action *can* be different to the argmax choice from the classifier. You should be sampling actions according to the probabilities from the classifier (a different sampling approach is possible in general, but won't work well with the rest of the instructions here, because we rely on seeing the different scores in correct ratios).
  3. Once you have some results with score functions - ideally from several runs to reduce correlation between inputs - then you can use them to construct a minibatch for supervised learning. Retrospectively set a bunch of $G\_t$ to match the $S\_t$ and $A\_t$ values.
  4. Calculate the "ground truth" for each input $S\_t$ as the action that was taken $A\_t$. It doesn't matter whether the action was the best or correct one to take, it matters that it was the one that was actually taken, and as above, ideally sampled according to your classifier probabilities.
  5. Calculate the parameter gradient for each input, output combination as normal for your neural network for supervised learning and multiclass cross-entropy loss . . . but
  6. Before you add to the total gradient for the minibatch, multiply the weights gradient by a normalised $G\_t$. You can change the normalisation parameters over time - typically you subtract some average $\bar{G}$ seen so far (e.g. running average of results from last 100 episodes) and maybe multiply by some value to put into an easy to handle range, so $\nabla\_{\theta, t} \leftarrow \beta(G\_t - \bar{G})\nabla\_{\theta, t}$. This is the "trick" from policy gradients that means the neural network will learn to avoid low total scores and find high total scores on average across many updates. The offset is not required, but makes the learning process faster and more stable. The multiplication is not required either, provided you are happy to try different learning rates instead. So the absolute bare bones version is $\nabla\_{\theta, t} \leftarrow G\_t\nabla\_{\theta, t}$
  7. Complete the minibatch by taking a positive gradient step on the mean gradient (perhaps multiplied by some learning rate, or modified with an optimiser like Adam)
  
  This is pretty much *REINFORCE with baseline*, a policy gradient method from RL, as applied to discrete action space and a softmax action classifier. It can in theory be applied in scenarios which are not technically setup as RL problems, as long as you can follow the procedure, and it reliably means something to generate a "good" sequence of output choices based on available inputs, by some measure (your score in this case).
  
  > **Comments:**
  > **username_0**: Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.  
  > **username_1**: @IAQuestions You should be able to vectorise the process described above. The difficult part is vectorising your environment and environmental responses, and typically that's impossible for MDPs. But if you have a way to vectorise assessing agent behaviour in your problem, all of steps 1-7 above are vectorisable. Steps 3-7 are vectorisable regardless and convert your scores to useful gradient.  
  
  ---
  
  