Answers.Body: <p>Short answer: Yes.</p>
  <p>Consider a non-linear regression on that dataset. Using a model of degree two, it would fit a quadratic exactly to your perfect data here. But I suppose you're asking about neural networks. You can have neural networks set up that are exactly equivalent to this kind of regression, so even with neural networks, yes you can get this non-linear extrapolation. Of course as you probably realise, you would have to know in advance what kind of behaviour you expect in this extrapolation before really trusting any extrapolated predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:35:31.347Z
Answers.LastActivityDate: 2021-04-09T05:43:46.930Z
Answers.LastEditDate: 2021-04-09T05:43:46.930Z
Answers.UserDisplayName: Cameron Chandler
Answers.Body: <p>It isn't too surprising to see behaviour like this, since you're using <span class="math-container">$\mathrm{ReLU}$</span> activation.</p>
  <p>Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.</p>
  <p><strong>Proposition</strong>. In a single-layer neural network with <span class="math-container">$n$</span> hidden neurons using <span class="math-container">$\mathrm{ReLU}$</span> activation, with one input and output node, the output is linear outside of the region <span class="math-container">$[A, B]$</span> for some <span class="math-container">$A &lt; B \in \mathbb{R}$</span>. In other words, if <span class="math-container">$x &gt; B$</span>, <span class="math-container">$f(x) = \alpha x + \beta$</span> for some constants <span class="math-container">$\alpha$</span> and <span class="math-container">$\beta$</span>, and if <span class="math-container">$x &lt; A$</span>, <span class="math-container">$f(x) = \gamma x + \delta$</span> for some constants <span class="math-container">$\gamma$</span> and <span class="math-container">$\delta$</span>.</p>
  <p><em>Proof.</em> I can write the neural network as a function <span class="math-container">$f \colon \mathbb R \to \mathbb R$</span>, defined by
  <span class="math-container">$$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$</span>
  Note that each neuron switches from being <span class="math-container">$0$</span> to a linear function, or vice versa, when <span class="math-container">$w_i x + b_i = 0$</span>. Define <span class="math-container">$r_i = -\frac{b_i}{w_i}$</span>. Then, I can set <span class="math-container">$B = \max_i r_i$</span> and <span class="math-container">$A = \min_i r_i$</span>. If <span class="math-container">$x &gt; B$</span>, each neuron will either be <span class="math-container">$0$</span> or linear, so <span class="math-container">$f$</span> is just a sum of linear functions, i.e. linear with constant gradient. The same applies if <span class="math-container">$x &lt; A$</span>.</p>
  <p>Hence, <span class="math-container">$f$</span> is a linear function with constant gradient if <span class="math-container">$x &lt; A$</span> or <span class="math-container">$x &gt; B$</span>.
  <span class="math-container">$\square$</span></p>
  <p>If the result isn't clear, here's an illustration of the idea:
  <a href="https://i.sstatic.net/7BtzI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7BtzI.png" alt="Plot of sum of ReLU" /></a></p>
  <p>This is a <span class="math-container">$3$</span>-neuron network, and I've marked the points I denote <span class="math-container">$r_i$</span> by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T14:35:10.277Z
Answers.Text: Am I misunderstanding this, or do different activation functions lead to the same result because they just activate the `y = weight*x + bias` linear function?
Answers.UserDisplayName: Mr. Demetrius Michael
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T14:59:29.397Z
Answers.Text: Well, you'd get slightly different results with other activations. As an example take a layer of neurons with sigmoid activation: you'd essentially have a function which is a sum of (scaled and repositioned) sigmoids. I haven't thought about it very deeply but I expect that as $x \to \infty$ you'd see each neuron saturate (either at zero or one) so the function would be "almost constant" after a certain point. Play around on a graph plotter with sums of [logistic functions](https://en.wikipedia.org/wiki/Logistic_function) to see what I mean, it's a fun exercise.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T08:52:00.143Z
Answers.LastActivityDate: 2021-04-07T08:52:00.143Z
Answers.UserDisplayName: htl
Body: <p>Training on a quadratic function</p>
  <pre><code>x = np.linspace(-10, 10, num=1000)
  np.random.shuffle(x)
  y = x**2
  </code></pre>
  <p>Will predict an expected quadratic curve between <code>-10 &lt; x &lt; 10</code>.</p>
  <p><a href="https://i.sstatic.net/vkkC4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vkkC4.png" alt="in bound" /></a></p>
  <p>Unfortunately my model's predictions become linear outside of the trained dataset.</p>
  <p>See <code>-100 &lt; x &lt; 100</code> below:</p>
  <p><a href="https://i.sstatic.net/mnlq5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mnlq5.png" alt="out of bounds" /></a></p>
  <p>Here is how I define my model:</p>
  <pre><code>model = keras.Sequential([
        layers.Dense(64, activation='relu'),
        layers.Dense(64, activation='relu'),
        layers.Dense(1)
    ])
  
  model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.1))
  
  history = model.fit(
      x, y,
      validation_split=0.2,
      verbose=0, epochs=100)
  </code></pre>
  <p><a href="https://colab.research.google.com/drive/1yPJY10Iu0XhWbJN6wOprtpYfNvauEiyl?usp=sharing" rel="nofollow noreferrer">Here's a link to a google colab for more context.</a></p>
  
Tags: <neural-networks><keras><regression>
Title: Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?
UserDisplayName: Mr. Demetrius Michael
ThreadText: # Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?
  
  (Asked by: username_0 on 2021-04-06)
  
  Training on a quadratic function
  
  ```
  x = np.linspace(-10, 10, num=1000)
  np.random.shuffle(x)
  y = x**2
  
  ```
  
  Will predict an expected quadratic curve between `-10 < x < 10`.
  
  [![in bound](https://i.sstatic.net/vkkC4.png)](https://i.sstatic.net/vkkC4.png)
  
  Unfortunately my model's predictions become linear outside of the trained dataset.
  
  See `-100 < x < 100` below:
  
  [![out of bounds](https://i.sstatic.net/mnlq5.png)](https://i.sstatic.net/mnlq5.png)
  
  Here is how I define my model:
  
  ```
  model = keras.Sequential([
        layers.Dense(64, activation='relu'),
        layers.Dense(64, activation='relu'),
        layers.Dense(1)
    ])
  
  model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.1))
  
  history = model.fit(
      x, y,
      validation_split=0.2,
      verbose=0, epochs=100)
  
  ```
  
  [Here's a link to a google colab for more context.](https://colab.research.google.com/drive/1yPJY10Iu0XhWbJN6wOprtpYfNvauEiyl?usp=sharing)
  
  **Tags:** neural-networks, keras, regression
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-04-07. Score: 3 | Accepted answer
  
  It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.
  
  Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.
  
  **Proposition**. In a single-layer neural network with $n$ hidden neurons using $\mathrm{ReLU}$ activation, with one input and output node, the output is linear outside of the region $[A, B]$ for some $A < B \in \mathbb{R}$. In other words, if $x > B$, $f(x) = \alpha x + \beta$ for some constants $\alpha$ and $\beta$, and if $x < A$, $f(x) = \gamma x + \delta$ for some constants $\gamma$ and $\delta$.
  
  *Proof.* I can write the neural network as a function $f \colon \mathbb R \to \mathbb R$, defined by
  $$f(x) = \sum\_{i = 1}^n \left[\sigma\_i\max(0, w\_i x + b\_i)\right] + c.$$
  Note that each neuron switches from being $0$ to a linear function, or vice versa, when $w\_i x + b\_i = 0$. Define $r\_i = -\frac{b\_i}{w\_i}$. Then, I can set $B = \max\_i r\_i$ and $A = \min\_i r\_i$. If $x > B$, each neuron will either be $0$ or linear, so $f$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $x < A$.
  
  Hence, $f$ is a linear function with constant gradient if $x < A$ or $x > B$.
  $\square$
  
  If the result isn't clear, here's an illustration of the idea:
  [![Plot of sum of ReLU](https://i.sstatic.net/7BtzI.png)](https://i.sstatic.net/7BtzI.png)
  
  This is a $3$-neuron network, and I've marked the points I denote $r\_i$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.
  
  > **Comments:**
  > **username_0**: Am I misunderstanding this, or do different activation functions lead to the same result because they just activate the `y = weight*x + bias` linear function?  
  > **username_2**: Well, you'd get slightly different results with other activations. As an example take a layer of neurons with sigmoid activation: you'd essentially have a function which is a sum of (scaled and repositioned) sigmoids. I haven't thought about it very deeply but I expect that as $x \to \infty$ you'd see each neuron saturate (either at zero or one) so the function would be "almost constant" after a certain point. Play around on a graph plotter with sums of [logistic functions](https://en.wikipedia.org/wiki/Logistic_function) to see what I mean, it's a fun exercise.  
  
  ---
  
  ### Answer by username_1 on 2021-04-07. Score: 1
  
  Short answer: Yes.
  
  Consider a non-linear regression on that dataset. Using a model of degree two, it would fit a quadratic exactly to your perfect data here. But I suppose you're asking about neural networks. You can have neural networks set up that are exactly equivalent to this kind of regression, so even with neural networks, yes you can get this non-linear extrapolation. Of course as you probably realise, you would have to know in advance what kind of behaviour you expect in this extrapolation before really trusting any extrapolated predictions.
  
  
  ---
  
  