Hi,
I am training a model using Pytorch and Pytorch Geometric and I’ve noticed a weird behavior of the loss function:

20K_samples2400×1200 96.2 KB

It seems that during training the network forgets everything. Moreover, the more samples I have in the dataset, the more often this kind of “reset” occurs (and when the dataset is large, the network doesn’t learn at all).
These are the optimizer and the loss function that I am using
criterion = nn.BCEWithLogitsLoss()

lr_start = 2e-4
l2_weight = 1e-10 # L2 regularization weight
optimizer = optim.Adam(net.parameters(), lr=lr_start, weight_decay=l2_weight)

lr_decay = torch.tensor(0.99, dtype=torch.float64)
lr_decay_steps = torch.tensor(5, dtype=torch.float64)
gamma = torch.pow(lr_decay, torch.div(1, lr_decay_steps))

scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=gamma, last_epoch=-1)

I am not an expert of neural network and Pytorch, so I was wondering if maybe there are some mistakes in the code above or what else could cause this issue…