Answers.Body: <p>Since you made no assumption on what <span class="math-container">$q$</span> is, the problem is untractable</p>
  <p>If you assume <span class="math-container">$q$</span> to be Gaussian, it's easy to prove that the KL distance is minimized when you have <span class="math-container">$\mu_{f(s)}, \sigma_{f(s)}$</span> equal to <span class="math-container">$\mu_q, \sigma_q$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T19:50:38.170Z
Answers.Text: but is there a closed-form or clever way to find s that $\mu_{f(s)},\sigma_{f(s)}$ equal to $\mu_q,\sigma_q$?
Answers.UserDisplayName: user3315463
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T21:36:09.563Z
Answers.Text: @user3315463 for the question you posted, it looks like you assume to have $q$, this implies that you also have $\mu,\sigma of $q$... if not, please add further details on what's $q$
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T18:45:15.333Z
Answers.LastActivityDate: 2024-04-25T18:45:15.333Z
Answers.UserDisplayName: Alberto
Body: <p>Consider a neural network function <span class="math-container">$f:\mathbb{R}\to distribution$</span>. For simplicity, maybe consider that it returns a gaussian distribution.</p>
  <p>I want to find <span class="math-container">$\arg\min_{s\in\mathbb{R}}D_{KL}(f(s),q)$</span> for some fixed distribution <span class="math-container">$q$</span>.</p>
  <p>Is there an efficient closed-form method to find such <span class="math-container">$s$</span>? or do I need to run gradient descent with respect to <span class="math-container">$s$</span> and it might get stuck into local-optima?</p>
  <p><br />
  <br />
  For someone who might want more context: I'm reading a paper about cross-domain transfer in RL.</p>
  <p>Cross-Domain Transfer via Semantic Skill Imitation, by Karl Pertsch and et.al...
  <a href="https://arxiv.org/pdf/2212.07407" rel="nofollow noreferrer">https://arxiv.org/pdf/2212.07407</a></p>
  <p>Here, to map a state in the source domain to a state in the target domain, the author suggests finding a target domain state that minimizes below loss. Here <span class="math-container">$p_S(k|s^S)$</span> is a distribution of semantics for the source state <span class="math-container">$s^S$</span>, and I think can be considered as fixed for each source state. <span class="math-container">$p_T(k|s^T)$</span> is a distribution of semantics for the target domain state <span class="math-container">$s^T$</span>(trained NN function). so minimizing this loss means we consider a target domain state that has the most similar semantics to the source domain state to be the representation of the source domain state in target domain.</p>
  <p><span class="math-container">$\min_{s^T \in D_T} \left( D_{KL}(p_T(k|s^T), p_S(k|s^S)) + D_{KL}(p_S(k|s^S), p_T(k|s^T)) \right)$</span></p>
  <p>However, as I know, state space is continuous, so I am confused how one can efficiently find such a target domain state that minimizes above KL term.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T16:15:58.630Z
Comments.Text: you made no assumption on $q$ thus, it's even very likely that you cannot even compute the KL (thus you can't even do GD)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T17:10:20.527Z
Comments.Text: @Alberto umm what if we consider q also to be a gaussian distribution?
Comments.UserDisplayName: user3315463
Tags: <deep-rl><optimization><transfer-learning><kl-divergence>
Title: How to find an argument of a NN function(which returns a distribution) to minimize a KL divergence?
UserDisplayName: user3315463
ThreadText: # How to find an argument of a NN function(which returns a distribution) to minimize a KL divergence?
  
  (Asked by: username_0 on 2024-04-24)
  
  Consider a neural network function $f:\mathbb{R}\to distribution$. For simplicity, maybe consider that it returns a gaussian distribution.
  
  I want to find $\arg\min\_{s\in\mathbb{R}}D\_{KL}(f(s),q)$ for some fixed distribution $q$.
  
  Is there an efficient closed-form method to find such $s$? or do I need to run gradient descent with respect to $s$ and it might get stuck into local-optima?
  
  For someone who might want more context: I'm reading a paper about cross-domain transfer in RL.
  
  Cross-Domain Transfer via Semantic Skill Imitation, by Karl Pertsch and et.al...
  <https://arxiv.org/pdf/2212.07407>
  
  Here, to map a state in the source domain to a state in the target domain, the author suggests finding a target domain state that minimizes below loss. Here $p\_S(k|s^S)$ is a distribution of semantics for the source state $s^S$, and I think can be considered as fixed for each source state. $p\_T(k|s^T)$ is a distribution of semantics for the target domain state $s^T$(trained NN function). so minimizing this loss means we consider a target domain state that has the most similar semantics to the source domain state to be the representation of the source domain state in target domain.
  
  $\min\_{s^T \in D\_T} \left( D\_{KL}(p\_T(k|s^T), p\_S(k|s^S)) + D\_{KL}(p\_S(k|s^S), p\_T(k|s^T)) \right)$
  
  However, as I know, state space is continuous, so I am confused how one can efficiently find such a target domain state that minimizes above KL term.
  
  **Tags:** deep-rl, optimization, transfer-learning, kl-divergence
  > **Comments:**
  > **username_1**: you made no assumption on $q$ thus, it's even very likely that you cannot even compute the KL (thus you can't even do GD)  
  > **username_0**: @Alberto umm what if we consider q also to be a gaussian distribution?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-25. Score: 0
  
  Since you made no assumption on what $q$ is, the problem is untractable
  
  If you assume $q$ to be Gaussian, it's easy to prove that the KL distance is minimized when you have $\mu\_{f(s)}, \sigma\_{f(s)}$ equal to $\mu\_q, \sigma\_q$
  
  > **Comments:**
  > **username_0**: but is there a closed-form or clever way to find s that $\mu_{f(s)},\sigma_{f(s)}$ equal to $\mu_q,\sigma_q$?  
  > **username_1**: @user3315463 for the question you posted, it looks like you assume to have $q$, this implies that you also have $\mu,\sigma of $q$... if not, please add further details on what's $q$  
  
  ---
  
  