Answers.Body: <p>You can even predict a sinusoid with much less than 20 samples: two previous samples suffice. The reason is that sinusoids appear as solutions of second-order linear difference equations.</p>
  <p>In other words, you can write the linear relationship of your sinusoid with frequency <span class="math-container">$f$</span> and phase <span class="math-container">$\phi$</span> as</p>
  <p><span class="math-container">$sin (f \times n + \phi) = a_1 sin (f \times (n-1) + \phi) + a_2 sin (f \times (n-2) + \phi)$</span></p>
  <p>with real prediction coefficients <span class="math-container">$a_1$</span> and <span class="math-container">$a_2$</span> and discrete time index <span class="math-container">$n$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T05:43:49.230Z
Answers.Text: So just to clarify, this only works for functions that are n-order linear differentiable with n being the lookback?
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T05:49:00.810Z
Answers.Text: Also, I can't find information on this anywhere, what do I need to search to learn more about how this works?
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T10:47:42.183Z
Answers.Text: I'm not sure you will find something on this specifically... This is more a result of understanding linear relationships and linear algebra well.  If you have done any research into sequences and series, this is typically a topic that is discussed, and modern linear algebra courses will usually discuss difference (and other) equations.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T11:26:22.263Z
Answers.Text: Maybe https://en.wikipedia.org/wiki/Linear_prediction is a good entry point.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T05:28:14.387Z
Answers.LastActivityDate: 2022-07-27T05:28:14.387Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>I have a neural network that's trained on a sine wave. It uses a lookback of 20 to see what the last 20 predictions were and predict the next value. This network has only a single Linear layer (input size 20, output size 1) with no activation function and from just that, it is able to extrapolate a sine wave almost perfectly.</p>
  <p>This is very confusing to me, as the <span class="math-container">$sin$</span> function is non-linear: <span class="math-container">$sin(a+b) \neq sin(a) + sin(b)$</span> so the network shouldn't be able to approximate it (very well).</p>
  <p>The code to reproduce this is below:</p>
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  
  import torch
  import torch.nn as nn
  from torch.utils.data import DataLoader, Dataset
  import torch.optim as optim
  
  X_train = np.arange(0,100,0.5) 
  y_train = np.sin(X_train)
  
  X_test = np.arange(100,200,0.5) 
  y_test = np.sin(X_test)
  
  n_features = 1
  
  train_series = torch.from_numpy(y_train)
  test_series = torch.from_numpy(y_test)
  
  # Expects input of (batch, sequence, features)
  # So shape should be (1, 179, 20) and labels (1, 1, 179)
  look_back = 20
  
  train_dataset = []
  train_labels = []
  for i in range(len(train_series)-look_back):
      train_dataset.append(train_series[i:i+20])
      train_labels.append(train_series[i+20])
  train_dataset = torch.stack(train_dataset).unsqueeze(0)
  train_labels = torch.stack(train_labels).unsqueeze(0).unsqueeze(2)
  
  class Net(nn.Module):
      def __init__(self, input_shape):
          super(Net, self).__init__()
          
          self.fc = nn.Linear(input_shape, 1)
      
      def forward(self, x):
          out = self.fc(x)
          return out
  
  
  model = Net(look_back).double()
  loss_function = nn.MSELoss()
  optimizer = optim.Adam(model.parameters(), lr=0.001)
  
  loss_curve = []
  for epoch in range(300):
      loss_total = 0
      
      model.zero_grad()
      
      predictions = model(train_dataset)
      
      loss = loss_function(predictions, train_labels)
      loss_total += loss.item()
      loss.backward()
      optimizer.step()
      loss_curve.append(loss_total)
  
  extrapolation = []
  seed_batch = test_series[:20].reshape(1, 1, 20)
  current_batch = seed_batch
  with torch.no_grad():
      for i in range(180):
          predicted_value = model(current_batch)
          extrapolation.append(predicted_value.item())
          current_batch = torch.cat((current_batch[:,:,1:], predicted_value), axis=2)
  
  x = np.arange(110,200,0.5)
  fig, ax = plt.subplots(1, 1, figsize=(15, 5))
  ax.plot(X_train,y_train, lw=2, label='train data')
  ax.plot(X_test,y_test, lw=3, c='y', label='test data')
  ax.plot(x,extrapolation, lw=3, c='r',linestyle = ':', label='extrapolation')
  ax.legend(loc=&quot;lower left&quot;)
  plt.show();
  </code></pre>
  <p>And it produces the following plot:
  <a href="https://i.sstatic.net/0j3oX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0j3oX.png" alt="Sine wave extrapolation" /></a></p>
  
Tags: <neural-networks><machine-learning><regression>
Title: Entirely linear neural network learning non-linear function
UserDisplayName: Recessive
ThreadText: # Entirely linear neural network learning non-linear function
  
  (Asked by: username_0 on 2022-07-27)
  
  I have a neural network that's trained on a sine wave. It uses a lookback of 20 to see what the last 20 predictions were and predict the next value. This network has only a single Linear layer (input size 20, output size 1) with no activation function and from just that, it is able to extrapolate a sine wave almost perfectly.
  This is very confusing to me, as the $sin$ function is non-linear: $sin(a+b) \neq sin(a) + sin(b)$ so the network shouldn't be able to approximate it (very well).
  The code to reproduce this is below:
  import numpy as np
  import matplotlib.pyplot as plt
  
  import torch
  import torch.nn as nn
  from torch.utils.data import DataLoader, Dataset
  import torch.optim as optim
  
  X_train = np.arange(0,100,0.5) 
  y_train = np.sin(X_train)
  
  X_test = np.arange(100,200,0.5) 
  y_test = np.sin(X_test)
  
  n_features = 1
  
  train_series = torch.from_numpy(y_train)
  test_series = torch.from_numpy(y_test)
  
  # Expects input of (batch, sequence, features)
  # So shape should be (1, 179, 20) and labels (1, 1, 179)
  look_back = 20
  
  train_dataset = []
  train_labels = []
  for i in range(len(train_series)-look_back):
      train_dataset.append(train_series[i:i+20])
      train_labels.append(train_series[i+20])
  train_dataset = torch.stack(train_dataset).unsqueeze(0)
  train_labels = torch.stack(train_labels).unsqueeze(0).unsqueeze(2)
  
  class Net(nn.Module):
      def __init__(self, input_shape):
          super(Net, self).__init__()
          
          self.fc = nn.Linear(input_shape, 1)
      
      def forward(self, x):
          out = self.fc(x)
          return out
  
  
  model = Net(look_back).double()
  loss_function = nn.MSELoss()
  optimizer = optim.Adam(model.parameters(), lr=0.001)
  
  loss_curve = []
  for epoch in range(300):
      loss_total = 0
      
      model.zero_grad()
      
      predictions = model(train_dataset)
      
      loss = loss_function(predictions, train_labels)
      loss_total += loss.item()
      loss.backward()
      optimizer.step()
      loss_curve.append(loss_total)
  
  extrapolation = []
  seed_batch = test_series[:20].reshape(1, 1, 20)
  current_batch = seed_batch
  with torch.no_grad():
      for i in range(180):
          predicted_value = model(current_batch)
          extrapolation.append(predicted_value.item())
          current_batch = torch.cat((current_batch[:,:,1:], predicted_value), axis=2)
  
  x = np.arange(110,200,0.5)
  fig, ax = plt.subplots(1, 1, figsize=(15, 5))
  ax.plot(X_train,y_train, lw=2, label='train data')
  ax.plot(X_test,y_test, lw=3, c='y', label='test data')
  ax.plot(x,extrapolation, lw=3, c='r',linestyle = ':', label='extrapolation')
  ax.legend(loc="lower left")
  plt.show();
  
  And it produces the following plot:
  
  **Tags:** neural-networks, machine-learning, regression
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-27. Score: 3 | Accepted answer
  
  You can even predict a sinusoid with much less than 20 samples: two previous samples suffice. The reason is that sinusoids appear as solutions of second-order linear difference equations.
  
  In other words, you can write the linear relationship of your sinusoid with frequency $f$ and phase $\phi$ as
  
  $sin (f \times n + \phi) = a\_1 sin (f \times (n-1) + \phi) + a\_2 sin (f \times (n-2) + \phi)$
  
  with real prediction coefficients $a\_1$ and $a\_2$ and discrete time index $n$.
  
  > **Comments:**
  > **username_0**: So just to clarify, this only works for functions that are n-order linear differentiable with n being the lookback?  
  > **username_0**: Also, I can't find information on this anywhere, what do I need to search to learn more about how this works?  
  > **username_2**: I'm not sure you will find something on this specifically... This is more a result of understanding linear relationships and linear algebra well.  If you have done any research into sequences and series, this is typically a topic that is discussed, and modern linear algebra courses will usually discuss difference (and other) equations.  
  > **username_1**: Maybe https://en.wikipedia.org/wiki/Linear_prediction is a good entry point.  
  
  ---
  
  