Body: <p>I'm working on a project where a non differentiable loss is there. I'm thinking about how should I deal with them.</p>
  <p>My model is a very big lstm model (about 1M parameter), and after 500 steps (not sure if it's enough) of <a href="https://en.m.wikipedia.org/wiki/Differential_evolution" rel="nofollow noreferrer">differential evolution</a>, the loss didn't decrease at all.</p>
  <p>I'm thinking that actor critic can see as follows:
  The policy network updates by value function, which is calculated from a reward estimator (Critic).</p>
  <p>I'm wondering if I can take the thought of &quot;estimator&quot;, and use it on loss estimation, then I can replace the non differentiable loss as that estimator, then also train the estimator each step. Maybe someone has already done that, if so, I want to know the name of the method.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-24T09:54:27.810Z
Comments.Text: what's your loss? what differential evolution method did you use? CE/CMA-ES?
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-24T10:36:53.033Z
Comments.Text: The part make the loss function not differentiable is "argmax" and "split the list by a special element". And I think DE is an algorithm just like cma-es.
Comments.UserDisplayName: TWTom
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-24T12:27:59.140Z
Comments.Text: i mean you can definitely try reinforce, not quite sure how to help without a formal definition of the loss
Comments.UserDisplayName: Alberto
Tags: <objective-functions><optimization><actor-critic-methods>
Title: Non differentiable loss function train with actor critic style
UserDisplayName: TWTom
ThreadText: # Non differentiable loss function train with actor critic style
  
  (Asked by: username_0 on 2024-01-24)
  
  I'm working on a project where a non differentiable loss is there. I'm thinking about how should I deal with them.
  
  My model is a very big lstm model (about 1M parameter), and after 500 steps (not sure if it's enough) of [differential evolution](https://en.m.wikipedia.org/wiki/Differential_evolution), the loss didn't decrease at all.
  
  I'm thinking that actor critic can see as follows:
  The policy network updates by value function, which is calculated from a reward estimator (Critic).
  
  I'm wondering if I can take the thought of "estimator", and use it on loss estimation, then I can replace the non differentiable loss as that estimator, then also train the estimator each step. Maybe someone has already done that, if so, I want to know the name of the method.
  
  **Tags:** objective-functions, optimization, actor-critic-methods
  > **Comments:**
  > **username_1**: what's your loss? what differential evolution method did you use? CE/CMA-ES?  
  > **username_0**: The part make the loss function not differentiable is "argmax" and "split the list by a special element". And I think DE is an algorithm just like cma-es.  
  > **username_1**: i mean you can definitely try reinforce, not quite sure how to help without a formal definition of the loss  
  
  ---
  
  ## 0 answers
  
  