Answers.Body: <p>A simpler answer is that for a standard neural net, the asymptotic behaviour is the asymptotic behaviour of the output neurons. For example, if the output layer is ReLUs, then the asymptotic behaviour is necessarily linear.</p>
  <p>In your case, since you want it to be asymptotically constant, you can use the slightly old-fashioned choice of sigmoid units in the output layer.</p>
  <p>The training set is necessarily finite, so no machine learning method can learn asymptotic behaviour. It can only be supplied by prior knowledge, for example as I describe.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T15:52:49.123Z
Answers.LastActivityDate: 2021-03-29T15:52:49.123Z
Answers.UserDisplayName: chrishmorris
Answers.Body: <p>For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.</p>
  <p><strong><a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3544698#:%7E:text=Instead%20of%20using%20the%20natural,the%20way%20to%20controlling%20asymptotics." rel="nofollow noreferrer">Neural Networks with Asymptotics Control</a></strong> discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on <strong>two novel technical constructs</strong>:</p>
  <ol>
  <li>A <strong>Multi-Dimensional Spline Interpolator</strong> with Prescribed Asymptotic Behavior</li>
  <li>and A <strong>Custom NN layer</strong> that guarantees zero asymptotics in chosen directions.</li>
  </ol>
  <p>Let <span class="math-container">$f(x)$</span>  be the Function that we want to approximate while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function <span class="math-container">$S(x)$</span> that has the same asymptotics as <span class="math-container">$f(x)$</span>.</p>
  <ul>
  <li>Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function <span class="math-container">$S(x)$</span>.
  <ul>
  <li><p>For step <strong>one</strong>, they show how to construct a universal control variate, a multi-dimensional spline <span class="math-container">$S(x)$</span> that has the same asymptotics as <span class="math-container">$f(x)$</span>; and that can be used in all situations. In specific applications, of course, if more is known about the function <span class="math-container">$f$</span>; a better choice of the control variate could be available.</p>
  </li>
  <li><p>For step <strong>two</strong>, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function <span class="math-container">$R(x) = S(x) - f(x)$</span> with a special NN that has vanishing (zero) asymptotics in all, or some, directions.</p>
  </li>
  </ul>
  </li>
  </ul>
  <p><strong>One-Dimensional Spline with Asymptotics Control:</strong></p>
  <p>The paper discussed that Instead of using the natural boundary conditions, i.e., <strong>setting second derivatives
  to zero at the boundaries</strong>, they have fixed the first derivatives at boundary points to
  arbitrary values. <span class="math-container">$\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$</span>.</p>
  <p>Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. <span class="math-container">$f(x) \simeq f_{−}(x)$</span> for <span class="math-container">$x&lt;h_0$</span> and <span class="math-container">$f(x) \simeq f_{+}(x)$</span>  for <span class="math-container">$x&gt;h_{N+1}$</span>
  for large negative <span class="math-container">$h_0$</span> and large positive <span class="math-container">$h_{N+1}$</span>. First, we
  expand the set of spline nodes to include points <span class="math-container">$h_0$</span> and <span class="math-container">$h_{N+1}$</span> and make sure
  the spline passes through them, <span class="math-container">$S(h_0) = f_{−}(h_0)$</span> and <span class="math-container">$S(h_{N+1}) = f_{+}(h_{N+1})$</span>. To
  finish, they have specifies first order derivatives at these new boundary points.</p>
  <p><span class="math-container">$$S^{\prime}_0(h_0) = f^{\prime}_{−}(h_0), S(h_{N+1}) = f^{\prime}_{+}(h_{N+1})$$</span></p>
  <p>This fully specifies the approximating function <span class="math-container">$\tilde{S}(x)$</span>,
  <span class="math-container">$$\tilde{S}(x)=\left\{\begin{array}{ll}f_{-}(x), &amp; x \leq h_{0} \\ S(x), &amp; h_{0}&lt;x&lt;h_{N+1} \\ f_{+}(x), &amp; h_{N+1} \leq x\end{array}\right.$$</span></p>
  <p><strong>One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives</strong></p>
  <p>where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between <span class="math-container">$h_0$</span> and <span class="math-container">$h_1$</span>, say <span class="math-container">$h_{\frac{1}{2}}$</span> , and find, analytically, the value <span class="math-container">$S(h_{\frac{1}{2}})$</span> such that the second derivative at <span class="math-container">$h_{0}, S^{\prime \prime}\left(h_{0}\right)= f_{-}^{\prime \prime}\left(h_{0}\right)$</span> is matched.</p>
  <blockquote>
  <p><strong><a href="https://www.researchgate.net/publication/280574583_Asymptotic_Behaviour_of_Gradient_Learning_Algorithms_in_Neural_Network_Models_for_the_Identification_of_Nonlinear_Systems" rel="nofollow noreferrer">Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems</a></strong>. This paper deals with studying the <strong>asymptotical properties</strong> of multilayer neural networks models used for the adaptive identification of the wide class of nonlinearly parameterized systems in a stochastic environment.</p>
  </blockquote>
  <blockquote>
  <p><strong><a href="https://ieeexplore.ieee.org/document/31317" rel="nofollow noreferrer">A study of the asymptotic behavior of neural networks</a></strong>. where they have studied neural networks modeled as a set of nonlinear differential equations of the form <span class="math-container">$TX+X=Wf(X)+b$</span>, where <span class="math-container">$X$</span> is the neural membrane potential vector, <span class="math-container">$W$</span> is the network connectivity matrix, and <span class="math-container">$F(X)$</span> is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity <span class="math-container">$W$</span> need not be symmetric. The simulated behavior of typical neural networks is presented</p>
  </blockquote>
  <p><strong>Refrence</strong></p>
  <p><a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3544698#:%7E:text=Instead%20of%20using%20the%20natural,the%20way%20to%20controlling%20asymptotics." rel="nofollow noreferrer">Neural Networks with Asymptotics Control</a></p>
  <p><a href="https://www.researchgate.net/publication/280574583_Asymptotic_Behaviour_of_Gradient_Learning_Algorithms_in_Neural_Network_Models_for_the_Identification_of_Nonlinear_Systems" rel="nofollow noreferrer">Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems</a></p>
  <p><a href="https://ieeexplore.ieee.org/document/31317" rel="nofollow noreferrer">A study of the asymptotic behavior of neural networks</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T13:21:43.737Z
Answers.LastActivityDate: 2021-09-29T07:09:10.283Z
Answers.LastEditDate: 2021-09-29T07:09:10.283Z
Answers.UserDisplayName: Faizy
Body: <p>Is it possible to specify what the asymptotic behaviour of a Neural Networks (NN) model should be?</p>
  <p>I am thinking of a NN which tries to learn a mapping <span class="math-container">$\vec y=f(\vec x)$</span> with <span class="math-container">$\vec x$</span> a vector of features of dimension <span class="math-container">$d$</span> and <span class="math-container">$\vec y$</span> a vector of outputs of dimension <span class="math-container">$p$</span>.</p>
  <p>Is it possible to specify that, for instance, the NN should have a fixed value when <span class="math-container">$x_1$</span> goes to infinite?</p>
  <p>I mean:
  <span class="math-container">$$
  \lim_{x_1\to \infty} f(\vec x) = \vec c
  $$</span></p>
  <p>If it is not possible with NN, do you know other machine learning models (for instance  Gaussian Process Regression or Support Vector Regression) which have a known asymptotic behaviour?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-21T20:31:12.993Z
Comments.Text: to gaurantee you would have to do some analysis on your network znd define the conditions of your network explicitly, but if you are okay with an approximation, you can define an auxillary loss $h(f(x,x_1=\alpha) - f(x,x_1=g(\alpha)))$ to try to make it fall in some form of algorithmic patter that will follow your needs
Comments.UserDisplayName: mshlis
Tags: <neural-networks><machine-learning><deep-learning><computational-learning-theory>
Title: Is it possible to control asymptotic behaviour of neural network models?
UserDisplayName: Ken Grimes
ThreadText: # Is it possible to control asymptotic behaviour of neural network models?
  
  (Asked by: username_0 on 2019-08-21)
  
  Is it possible to specify what the asymptotic behaviour of a Neural Networks (NN) model should be?
  
  I am thinking of a NN which tries to learn a mapping $\vec y=f(\vec x)$ with $\vec x$ a vector of features of dimension $d$ and $\vec y$ a vector of outputs of dimension $p$.
  
  Is it possible to specify that, for instance, the NN should have a fixed value when $x\_1$ goes to infinite?
  
  I mean:
  $$
  \lim\_{x\_1\to \infty} f(\vec x) = \vec c
  $$
  
  If it is not possible with NN, do you know other machine learning models (for instance Gaussian Process Regression or Support Vector Regression) which have a known asymptotic behaviour?
  
  **Tags:** neural-networks, machine-learning, deep-learning, computational-learning-theory
  
  > **Comments:**
  > **username_1**: to gaurantee you would have to do some analysis on your network znd define the conditions of your network explicitly, but if you are okay with an approximation, you can define an auxillary loss $h(f(x,x_1=\alpha) - f(x,x_1=g(\alpha)))$ to try to make it fall in some form of algorithmic patter that will follow your needs  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-03-29. Score: 1
  
  A simpler answer is that for a standard neural net, the asymptotic behaviour is the asymptotic behaviour of the output neurons. For example, if the output layer is ReLUs, then the asymptotic behaviour is necessarily linear.
  
  In your case, since you want it to be asymptotically constant, you can use the slightly old-fashioned choice of sigmoid units in the output layer.
  
  The training set is necessarily finite, so no machine learning method can learn asymptotic behaviour. It can only be supplied by prior knowledge, for example as I describe.
  
  
  ---
  
  ### Answer by username_3 on 2021-03-29. Score: 1
  
  For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.
  
  **[Neural Networks with Asymptotics Control](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3544698#:%7E:text=Instead%20of%20using%20the%20natural,the%20way%20to%20controlling%20asymptotics.)** discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on **two novel technical constructs**:
  
  1. A **Multi-Dimensional Spline Interpolator** with Prescribed Asymptotic Behavior
  2. and A **Custom NN layer** that guarantees zero asymptotics in chosen directions.
  
  Let $f(x)$ be the Function that we want to approximate while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function $S(x)$ that has the same asymptotics as $f(x)$.
  
  * Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function $S(x)$.
    + For step **one**, they show how to construct a universal control variate, a multi-dimensional spline $S(x)$ that has the same asymptotics as $f(x)$; and that can be used in all situations. In specific applications, of course, if more is known about the function $f$; a better choice of the control variate could be available.
    + For step **two**, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function $R(x) = S(x) - f(x)$ with a special NN that has vanishing (zero) asymptotics in all, or some, directions.
  
  **One-Dimensional Spline with Asymptotics Control:**
  
  The paper discussed that Instead of using the natural boundary conditions, i.e., **setting second derivatives
  to zero at the boundaries**, they have fixed the first derivatives at boundary points to
  arbitrary values. $\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$.
  
  Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. $f(x) \simeq f\_{−}(x)$ for $x<h\_0$ and $f(x) \simeq f\_{+}(x)$ for $x>h\_{N+1}$
  for large negative $h\_0$ and large positive $h\_{N+1}$. First, we
  expand the set of spline nodes to include points $h\_0$ and $h\_{N+1}$ and make sure
  the spline passes through them, $S(h\_0) = f\_{−}(h\_0)$ and $S(h\_{N+1}) = f\_{+}(h\_{N+1})$. To
  finish, they have specifies first order derivatives at these new boundary points.
  
  $$S^{\prime}\_0(h\_0) = f^{\prime}\_{−}(h\_0), S(h\_{N+1}) = f^{\prime}\_{+}(h\_{N+1})$$
  
  This fully specifies the approximating function $\tilde{S}(x)$,
  $$\tilde{S}(x)=\left\{\begin{array}{ll}f\_{-}(x), & x \leq h\_{0} \\ S(x), & h\_{0}<x<h\_{N+1} \\ f\_{+}(x), & h\_{N+1} \leq x\end{array}\right.$$
  
  **One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives**
  
  where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between $h\_0$ and $h\_1$, say $h\_{\frac{1}{2}}$ , and find, analytically, the value $S(h\_{\frac{1}{2}})$ such that the second derivative at $h\_{0}, S^{\prime \prime}\left(h\_{0}\right)= f\_{-}^{\prime \prime}\left(h\_{0}\right)$ is matched.
  
  > **[Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems](https://www.researchgate.net/publication/280574583_Asymptotic_Behaviour_of_Gradient_Learning_Algorithms_in_Neural_Network_Models_for_the_Identification_of_Nonlinear_Systems)**. This paper deals with studying the **asymptotical properties** of multilayer neural networks models used for the adaptive identification of the wide class of nonlinearly parameterized systems in a stochastic environment.
  
  > **[A study of the asymptotic behavior of neural networks](https://ieeexplore.ieee.org/document/31317)**. where they have studied neural networks modeled as a set of nonlinear differential equations of the form $TX+X=Wf(X)+b$, where $X$ is the neural membrane potential vector, $W$ is the network connectivity matrix, and $F(X)$ is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity $W$ need not be symmetric. The simulated behavior of typical neural networks is presented
  
  **Refrence**
  
  [Neural Networks with Asymptotics Control](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3544698#:%7E:text=Instead%20of%20using%20the%20natural,the%20way%20to%20controlling%20asymptotics.)
  
  [Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems](https://www.researchgate.net/publication/280574583_Asymptotic_Behaviour_of_Gradient_Learning_Algorithms_in_Neural_Network_Models_for_the_Identification_of_Nonlinear_Systems)
  
  [A study of the asymptotic behavior of neural networks](https://ieeexplore.ieee.org/document/31317)
  
  
  ---
  
  