Body: <p>I am trying to learn a rather complicated ODE with a neural network. However, I wanted to start simple and looked at the most basic (yet not trivial) ODE, namely <span class="math-container">$y' = y$</span> with initial value <span class="math-container">$y(0) = 1$</span>.</p>
  <p>I used the following code:</p>
  <pre><code>import numpy as np
  import torch
  from torch import nn
  import matplotlib.pyplot as plt
  
  class neural_net(nn.Module):
      
      def __init__(self):
          super().__init__()
  
          self.network = nn.Sequential(
              nn.Linear(1, 64),
              nn.Tanh(),
              nn.Linear(64, 64),
              nn.Tanh(),
              nn.Linear(64, 64),
              nn.Tanh(),
              nn.Linear(64, 1),
          )
  
          self.apply(self._init_weights)
      
      def _init_weights(self, module):
          if isinstance(module, nn.Linear):
              module.weight.data.normal_(mean=0, std=1.0)
              module.bias.data.normal_(mean=0, std=1.0)
  
      
      def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
          return self.network(x)
  
  initial_value = 1
  model = neural_net()
  
  # choose grid
  S = 0
  T = 1
  N = 200
  t = torch.tensor(np.linspace(S, T, N), requires_grad=True, dtype=torch.float).unsqueeze(1)
  
  # optimizer
  optimizer = torch.optim.Adam(params = model.parameters(), lr = 0.001)
  
  epochs = 4000
  loss_values = []
  for epoch in range(epochs):
  
      # activate training mode
      model.train()
  
      # 1. Forward pass
      y_pred = model(t)
      
      # 2. get derivative of model and calculate loss
      derivative = torch.autograd.grad(outputs=y_pred, inputs=t, grad_outputs=torch.ones_like(y_pred), create_graph=True)[0]
      loss = torch.mean((1*y_pred - derivative)**2) + (y_pred[0] - initial_value)**2
  
      # 3. Optimizer zero grad
      optimizer.zero_grad()
  
      # 4. Backward pass
      loss.backward()
  
      # 5. Optimizer step
      optimizer.step()
  
      # deactivate training mode
      model.eval()
      loss_values.append(loss.item())
      if epoch % 1000 == 0: print(f&quot;Epoch : {epoch} | Loss : {np.round(loss.item(), 4)}&quot;)
  
  plt.plot(loss_values)
  
  </code></pre>
  <p>And actually this works quite well. However, my problem is now the following: when I change the ODE to be solved to <span class="math-container">$y' = 10y$</span> the loss does still converge to some value, but not to zero. This means, that solving the ODE with my neural net fails.</p>
  <p>I get that the target function grows much more rapidly, but I am not sure which thing needs change in order to still get convergence to zero. I tried experimenting with the learning rate, the time grid, initialising the weights differently, longer training... . But nothing seems to work for me. Do you have any suggestions what I could try?</p>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-12T20:51:36.907Z
Comments.Text: Without BatchNorm, Adam optimizer (and many other optimizers) typically benefits from having the features and labels scaled similarly for stable and efficient training. Try scaling label using logarithm transformation, for example.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-13T10:25:15.460Z
Comments.Text: Thanks, for the reply. So you mean, to log-transform the ODE? This would be rather easy I guess, since the net has to learn only $z \to 10z$. I guess this might not work for the more complicated ODE. Or did you mean something else?
Comments.UserDisplayName: julian2000P
Tags: <neural-networks><math>
Title: How to learn the solution of an ODE using a neural network?
UserDisplayName: julian2000P
ThreadText: # How to learn the solution of an ODE using a neural network?
  
  (Asked by: username_0 on 2024-12-12)
  
  I am trying to learn a rather complicated ODE with a neural network. However, I wanted to start simple and looked at the most basic (yet not trivial) ODE, namely $y' = y$ with initial value $y(0) = 1$.
  I used the following code:
  import numpy as np
  import torch
  from torch import nn
  import matplotlib.pyplot as plt
  
  class neural_net(nn.Module):
      
      def __init__(self):
          super().__init__()
  
          self.network = nn.Sequential(
              nn.Linear(1, 64),
              nn.Tanh(),
              nn.Linear(64, 64),
              nn.Tanh(),
              nn.Linear(64, 64),
              nn.Tanh(),
              nn.Linear(64, 1),
          )
  
          self.apply(self._init_weights)
      
      def _init_weights(self, module):
          if isinstance(module, nn.Linear):
              module.weight.data.normal_(mean=0, std=1.0)
              module.bias.data.normal_(mean=0, std=1.0)
  
      
      def forward(self, x: torch.Tensor) -> torch.Tensor:
          return self.network(x)
  
  initial_value = 1
  model = neural_net()
  
  # choose grid
  S = 0
  T = 1
  N = 200
  t = torch.tensor(np.linspace(S, T, N), requires_grad=True, dtype=torch.float).unsqueeze(1)
  
  # optimizer
  optimizer = torch.optim.Adam(params = model.parameters(), lr = 0.001)
  
  epochs = 4000
  loss_values = []
  for epoch in range(epochs):
  
      # activate training mode
      model.train()
  
      # 1. Forward pass
      y_pred = model(t)
      
      # 2. get derivative of model and calculate loss
      derivative = torch.autograd.grad(outputs=y_pred, inputs=t, grad_outputs=torch.ones_like(y_pred), create_graph=True)[0]
      loss = torch.mean((1*y_pred - derivative)**2) + (y_pred[0] - initial_value)**2
  
      # 3. Optimizer zero grad
      optimizer.zero_grad()
  
      # 4. Backward pass
      loss.backward()
  
      # 5. Optimizer step
      optimizer.step()
  
      # deactivate training mode
      model.eval()
      loss_values.append(loss.item())
      if epoch % 1000 == 0: print(f"Epoch : {epoch} | Loss : {np.round(loss.item(), 4)}")
  
  plt.plot(loss_values)
  
  
  And actually this works quite well. However, my problem is now the following: when I change the ODE to be solved to $y' = 10y$ the loss does still converge to some value, but not to zero. This means, that solving the ODE with my neural net fails.
  I get that the target function grows much more rapidly, but I am not sure which thing needs change in order to still get convergence to zero. I tried experimenting with the learning rate, the time grid, initialising the weights differently, longer training... . But nothing seems to work for me. Do you have any suggestions what I could try?
  Thanks in advance!
  
  **Tags:** neural-networks, math
  > **Comments:**
  > **username_1**: Without BatchNorm, Adam optimizer (and many other optimizers) typically benefits from having the features and labels scaled similarly for stable and efficient training. Try scaling label using logarithm transformation, for example.  
  > **username_0**: Thanks, for the reply. So you mean, to log-transform the ODE? This would be rather easy I guess, since the net has to learn only $z \to 10z$. I guess this might not work for the more complicated ODE. Or did you mean something else?  
  
  ---
  
  ## 0 answers
  
  