Hello everyone,
I have controlled Stochastic process X_t and I am trying to use the method 1(P.14) developped in Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II – The Finite Horizon Case to minimize the loss associated to the control. The main idea is that I will consider the control to be a fonction of my process X_t and i will try to approximate this function with a neural network by applying the SGD to a loss function.
My first concern is that the article states that we shall assume that the activation function is 2pi-periodic but there does not seems to be a practical 2pi-periodic activation function. (I used the reLu function in my code).
Secondly, the loss function is approximatied by simulating N times the process Xt0,…,X_t_nt where t0,…,tnt is the time grid.
The problem is that I try to plot the loss computed at each iteration but it does not decrease. My guess is that the gradient computed during the .backward() is wrong but i do not understand why. Also, i have Problem understanding where should i put requires_grad = True. I have rewriten my code many times but i still can’t really understand what is wrong with my code.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from time import time
import numpy as np
from math import *
import matplotlib.pyplot as plt
from progress.bar import Bar


device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#nn with 2 Hidden layers, relu activation function
class nets(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 50)
        self.layer2 = nn.Linear(50, 50)
        self.layer3 = nn.Linear(50, 1)

    def forward(self, x):
        #on utilise la fonction relu pour les deux fonctions d'activations
        y = self.layer1(x)
        y = torch.relu(y)
        y = self.layer2(y)
        y = torch.relu(y)
        y = self.layer3(y)
        return y

#this function returns a Tensor where each row correspond to a simulation 
#and each colons to a time step
def env_simple(T, N, Nt):
    dt = T/Nt
    X = torch.zeros((N,Nt+1), device = device)
    A = torch.zeros((N,Nt+1), device = device)
    for i in range(N):
        for j in range(Nt+1):
            A[i,j] = rdn.forward( torch.tensor([ 0,X[i,j] ],dtype=torch.float) )
            if (j == 0):
                X[i,j] = rdn.forward( torch.tensor([ 0,X[i,j] ],dtype=torch.float) )
            else:
                dW = torch.normal(mean = 0.0, std = sqrt(dt), size = (1,1))
#dXt = alpha_t*d_t + dW_t
                #X_t{i+1} = X_ti + phi_theta( ti, X_ti)*deltaT + deltaW
                X[i,j] = X[i,j-1] + A[i,j-1]*(dt)+dW
                A[i,j] = rdn.forward( torch.tensor([ dt*j,X[i,j] ],dtype=torch.float) )
    return X, A

#f(X_t, alpha_t) = X_t**2 + alpha_t**2 + E[alpha_t]**2
def loss_fn(X, A, N):
    return (X.pow(2).sum()/n)+(A.pow(2).sum()/n) + (A.pow(2).sum()/n**2)


rdn = nets()
rdn.to(device)
optimizer = optim.SGD(params = rdn.parameters(), lr = 0.001, momentum = 0.9)

#rdn.requires_grad_(True)
m, n, nt = 10000, 1000, 10
T = 1
dt = T/nt
bar = Bar('Processing', max=m)
cout = []
for pp in range(m) :
    x, a = env_simple(T, n, nt)
    loss = loss_fn(x, a, n)
    cout.append(loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    bar.next()
bar.finish()

plt.plot(cout)
plt.show()

Thank you for the taking the time to help me!
Edit: since I am trying to get the grad wrt to rdn.parameters() I changed the line:
loss.backward() 

by:
loss.backward(rdn.parameters()) 

but i got this error instead:
  File "<tmp 6>", line 75, in <module>
    loss.backward(rdn.parameters())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/autograd/__init__.py", line 143, in backward
    grad_tensors_ = _make_grads(tensors, grad_tensors_)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/autograd/__init__.py", line 33, in _make_grads
    raise RuntimeError("Mismatch in shape: grad_output["
RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([50, 2]) and output[0] has a shape of torch.Size([]).