I’m very new to pytorch and I’m very stuck with model converging. It seems to me it is not learning since the loss/r2 do not improve.
What I’ve checked/tried based on the suggestions I found here.

changes/wrote from scratch loss function
set “loss.requires_grad = True”
tried to feed the data without dataloader / just straight manual batches
played with 2d data / mean pooled data!!! I got decent results for mean pooled data in Random Forest and SVM regressor, but not in NN, which confuses me and makes me think that the data is OK and the net is NOT ok!
played with learning rate, batch size

etc.
About the data: input is bert embeddings from letter-sequence, each data point is 1024 features 43 rows (for Conv1d I transpose it to 1024*43) Total >40K data point in train, batch size=64
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F ## relu, tahn
import torch.utils.data as DataLoader # helps create batches to train on
from scipy.stats import pearsonr
import numpy as np
import torch.utils.data as data_utils
torch.set_printoptions(precision=10)


#Hyperparameters
learning_rate=0.001
batch_size = 64
num_epochs=100


data_train1 = torch.Tensor(data_train)
targets_train1=torch.Tensor(targets_train)

dataset_train = data_utils.TensorDataset(data_train1, targets_train1)
train_loader = DataLoader.DataLoader(dataset=dataset_train, batch_size=batch_size, shuffle=True)


class NN (nn.Module):
    def __init__(self):#input_size=43x1024
        super(NN,self).__init__()
        self.layers = nn.Sequential(
            nn.Conv1d(1024, 512, kernel_size=4), #I tried different in and out here
            nn.ELU(),
            nn.BatchNorm1d(512),
            nn.Flatten(),
            nn.Linear(512*40, 512),
            nn.ReLU(),
            nn.Linear(512, 1)
        )
        
    def forward(self, x):
        return self.layers(x)

torch.manual_seed(100)

#Initialize network
model=NN().to(device)

#Loss and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate) #to check 


#Training the model
metric_name='r2'

for epoch in range(num_epochs):
    score=[]
    loss_all=[]
    print(f"Epoch: {epoch+1}/{num_epochs}")

    model.train()

    for batch_idx, (data, targets) in enumerate(train_loader):
        data=data.to(device=device)
        targets=targets.to(device=device)

        optimizer.zero_grad()
        
        #forward
        predictions=model(data)
        
        loss=criterion(predictions,targets).to(device)

        loss.requires_grad = True

        #backward
        loss_all.append(loss.item())
        loss.backward()
    
        #gradient descent or adam step
        optimizer.step()
        
        #computing r squared
        output=predictions.detach().cpu().numpy()
        target=targets.detach().cpu().numpy()
        output=np.squeeze(output)
        target=np.squeeze(target)
    
        score.append(pearsonr(target, output)[0]**2)

    total_score = sum(score)/len(score)
    print(f'training {metric_name}: {total_score}, mean loss: {sum(loss_all)/len(loss_all)}')

Output for (10) first Epochs:
Epoch: 1/100 training r2: 0.0026224905802415955, mean loss: 0.5084380856556941
Epoch: 2/100 training r2: 0.0026334153423518466, mean loss: 0.5082988155293148
Epoch: 3/100 training r2: 0.002577073836564485, mean loss: 0.5085703951569392
Epoch: 4/100 training r2: 0.002633483899689855, mean loss: 0.5081870414129565
Epoch: 5/100 training r2: 0.0025642136678393776, mean loss: 0.5083346445680192
Epoch: 6/100 training r2: 0.0026261540869286933, mean loss: 0.5084220717277274
Epoch: 7/100 training r2: 0.002614604670602339, mean loss: 0.5082813335398275
Epoch: 8/100 training r2: 0.0024826257263258784, mean loss: 0.5086268588042153
Epoch: 9/100 training r2: 0.00261018096876641, mean loss: 0.5082496945227619
Epoch: 10/100 training r2: 0.002542892071836945, mean loss: 0.5088265852086478
Response is float64 in range (-2,2).
With response scaled to float64 in range [-1,1] and tanh still not converging. I feel like something general is missing. BTW, when I do non shuffled batches and straight-forward data for batches (first batch with indices 0-63, second bath 64-127 etc) I get the same score results in each of epochs!
Tried to add (2) more sequences with Conv1d, BatchNorm1d, ELU (1024->512, 512->256, 256->128, kernel size 4) and the result is worse:( it is not learning at all!
I also tried to do mean pooled for data to have only 1024 for the input, with the same poor result for NN.
For me it seems like I’m missing something very general and the model just do not learn.
Hope you can help! Thank you in advance for your time!