Body: <p>Consider a prediction problem for example. This is a loss function (negative log likelihood) that I am roughly talking about:</p>
  
  <p><span class="math-container">\begin{align*}
      J_{\text{train}} &amp;= -\sum_t \log L\left(\theta_t, X_t, Y_t\right) \\
      J_{\text{test}} &amp;= -\sum_t \log L\left(\theta_t, X_{t+1}, Y_{t+1}\right) \\
      J_{\text{dyn}} &amp;= - \sum_t \frac{\left(\theta_{t+1} - \text{stop_gradient}({\theta_t})\right)^2}{2 \sigma_{\theta}^2} \\
      \theta_0 &amp;\sim \Psi
  \end{align*}</span></p>
  
  <p>Here, the dynamics are simply first-order smoothness (Brownian motion) in the time-dependent parameters <span class="math-container">$\text{d}\theta_t = \sigma_{\theta} \text{d} W_t$</span>.</p>
  
  <p>This is basically what they do in weather systems and in papers like <a href="http://www.hairer.org/papers/bayesian.pdf" rel="nofollow noreferrer">A Bayesian Approach to Data Assimilation</a>.</p>
  
  <p>The main difference is, we now have the ability to use stop gradients in the smoothness, which changes the problem somewhat. Other issues might arise.</p>
  
  <p>Who else is doing this? Feel free to also provide links to papers on related work. I am not finding enough hits, so maybe I am missing something. Are RNNs, if twisted slightly, falling into this framework and I am no seeing it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-01T12:50:35.997Z
Comments.Text: Please, provide more context about your initial equations. Where did you find them and in which context?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-01T15:44:47.690Z
Comments.Text: I created them. There is literally nothing else needed for the formulation but the context is, for example, minimize the negative log likelihood where you have a model for observations. The dynamics log likelihood is a model for the dynamics. 
  
  This is typical in data assimilation (see the paper above) but I have not found people using stop_gradient style constructions.
Comments.UserDisplayName: safetyduck
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T17:54:49.117Z
Comments.Text: @nbro Not sure if that update went through.
Comments.UserDisplayName: safetyduck
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T01:07:00.310Z
Comments.Text: I will _maybe_ look at your question later. Meanwhile, you could start a [bounty](https://meta.stackexchange.com/questions/16065/how-does-the-bounty-system-work) to attract attention, but you will lose reputation!
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning>
Title: Where can I find people solving this smoothing, filtering, temporal learning problem?
UserDisplayName: safetyduck
ThreadText: # Where can I find people solving this smoothing, filtering, temporal learning problem?
  
  (Asked by: username_0 on 2020-03-01)
  
  Consider a prediction problem for example. This is a loss function (negative log likelihood) that I am roughly talking about:
  
  \begin{align\*}
  J\_{\text{train}} &= -\sum\_t \log L\left(\theta\_t, X\_t, Y\_t\right) \\
  J\_{\text{test}} &= -\sum\_t \log L\left(\theta\_t, X\_{t+1}, Y\_{t+1}\right) \\
  J\_{\text{dyn}} &= - \sum\_t \frac{\left(\theta\_{t+1} - \text{stop\_gradient}({\theta\_t})\right)^2}{2 \sigma\_{\theta}^2} \\
  \theta\_0 &\sim \Psi
  \end{align\*}
  
  Here, the dynamics are simply first-order smoothness (Brownian motion) in the time-dependent parameters $\text{d}\theta\_t = \sigma\_{\theta} \text{d} W\_t$.
  
  This is basically what they do in weather systems and in papers like [A Bayesian Approach to Data Assimilation](http://www.hairer.org/papers/bayesian.pdf).
  
  The main difference is, we now have the ability to use stop gradients in the smoothness, which changes the problem somewhat. Other issues might arise.
  
  Who else is doing this? Feel free to also provide links to papers on related work. I am not finding enough hits, so maybe I am missing something. Are RNNs, if twisted slightly, falling into this framework and I am no seeing it?
  
  **Tags:** neural-networks, machine-learning
  > **Comments:**
  > **username_1**: Please, provide more context about your initial equations. Where did you find them and in which context?  
  > **username_0**: I created them. There is literally nothing else needed for the formulation but the context is, for example, minimize the negative log likelihood where you have a model for observations. The dynamics log likelihood is a model for the dynamics.   This is typical in data assimilation (see the paper above) but I have not found people using stop_gradient style constructions.  
  > **username_0**: @nbro Not sure if that update went through.  
  > **username_1**: I will _maybe_ look at your question later. Meanwhile, you could start a [bounty](https://meta.stackexchange.com/questions/16065/how-does-the-bounty-system-work) to attract attention, but you will lose reputation!  
  
  ---
  
  ## 0 answers
  
  