Hello,
I’m new to pytorch and run into first problem right away and hope to get some help here.
So this is my data generating function:
n_samples = 100
X = np.random.normal(size=(n_samples, 1))
y = np.random.normal(np.cos(5.*X) / (np.abs(X) + 1.), 0.1).ravel()
X_pred = np.atleast_2d(np.linspace(-3., 3., num=100)).T
X = np.hstack((X, X**2, X**3))
X_pred = np.hstack((X_pred, X_pred**2, X_pred**3))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, y_train, X_test, y_test, X_pred = map(torch.tensor, (X_train, y_train, X_test, y_test, X_pred))
train_ds = TensorDataset(X_train, y_train)
train_dl = DataLoader(train_ds, batch_size=6)

My first neural network looks like this:
input_dim = 3    
hidden_dim = 100  
output_dim = 1    

model = torch.nn.Sequential(
        torch.nn.Dropout(p=0.1), 
        torch.nn.Linear(input_dim, hidden_dim),
        torch.nn.ReLU(),
        torch.nn.Dropout(p=0.1),
        torch.nn.Linear(hidden_dim, hidden_dim),
        torch.nn.ReLU(),
        torch.nn.Dropout(p=0.1),
        torch.nn.Linear(hidden_dim, output_dim),
)  

loss_fn = torch.nn.MSELoss()
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=0.004)

This is m training loop:
for epoch in range(40):  

    
    for i, data in enumerate(train_dl, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs.float())
        loss = loss_fn(outputs, labels.float())
        loss.backward()
        optimizer.step()

        # print statistics
    if epoch % 10 == 0:
        print(loss_fn(outputs, labels.float()))
print(loss_fn(outputs, labels.float()))

print('Finished Training')


My problem is that the network doesn’t fit at all. When i plot the results it is just a straight line. I tried for hours tune different parameters and look for errors in my code but unfortunatly I did not come to better result.
Hope someone has an idea or hint what do change in the model.