Body: <p>Suppose we have a neural network <span class="math-container">$f_\theta(x)$</span>, where <span class="math-container">$x$</span> is the input and <span class="math-container">$\theta$</span> is the network's parameters.</p>
  <p>For each <span class="math-container">$\theta$</span>, we can minimize <span class="math-container">$f_\theta(x)$</span> w.r.t. <span class="math-container">$x$</span> and obtain the minimum point <span class="math-container">$x^*(\theta):=\arg\min_x f_\theta(x)$</span>.</p>
  <p>My question is: how to estimate the gradient: <span class="math-container">$\frac{d x^*(\theta)}{d \theta}$</span> ?</p>
  <p>I did some surveys and found this gradient (in fact any gradient) can be estimated by the <a href="https://en.wikipedia.org/wiki/Finite_difference" rel="nofollow noreferrer">finite difference</a> technique. However, it is rather slow when <span class="math-container">$\theta$</span> includes many parameters, <em>e.g.</em>, <span class="math-container">$f_\theta$</span> is a large neural network.</p>
  <p>Any possible approach is welcomed.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-20T11:51:46.433Z
Comments.Text: Argmin is not differentiable, so what gradient are you trying to estimate? It does not actually exist.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-20T12:05:17.190Z
Comments.Text: As far as I know, argmin is not differentiable means: for $x^*=\arg\min x$, $\frac{dx^*}{dx}$ is either $0$ or does not exist. However, here, I am interested in the gradient $\frac{dx^*}{d\theta}$. I am not sure whether this gradient exist. Any suggestion?
Comments.UserDisplayName: Mingzhou Liu
Tags: <neural-networks><optimization><gradient>
Title: How to estimate the gradient of an argmin loss
UserDisplayName: Mingzhou Liu
ThreadText: # How to estimate the gradient of an argmin loss
  
  (Asked by: username_0 on 2023-02-20)
  
  Suppose we have a neural network $f\_\theta(x)$, where $x$ is the input and $\theta$ is the network's parameters.
  
  For each $\theta$, we can minimize $f\_\theta(x)$ w.r.t. $x$ and obtain the minimum point $x^\*(\theta):=\arg\min\_x f\_\theta(x)$.
  
  My question is: how to estimate the gradient: $\frac{d x^\*(\theta)}{d \theta}$ ?
  
  I did some surveys and found this gradient (in fact any gradient) can be estimated by the [finite difference](https://en.wikipedia.org/wiki/Finite_difference) technique. However, it is rather slow when $\theta$ includes many parameters, *e.g.*, $f\_\theta$ is a large neural network.
  
  Any possible approach is welcomed.
  
  **Tags:** neural-networks, optimization, gradient
  
  > **Comments:**
  > **username_1**: Argmin is not differentiable, so what gradient are you trying to estimate? It does not actually exist.  
  > **username_0**: As far as I know, argmin is not differentiable means: for $x^*=\arg\min x$, $\frac{dx^*}{dx}$ is either $0$ or does not exist. However, here, I am interested in the gradient $\frac{dx^*}{d\theta}$. I am not sure whether this gradient exist. Any suggestion?  
  
  ---
  
  ## 0 answers
  
  