What do your loss and accuracy curves look like?
Some potential issues:


optimizer

make sure to pass all of your model’s parameters to the optimizer: optimizer = torch.optim.SGD(model.parameters(), lr = 0.01, momentum=0.9) as an example.
make sure to clear the optimizer at the start of running each batch: optimizer.zero_grad()




hyperparameters

make sure your hyperparameters have an appropriate size: learning rate, batch size, weight decay, etc, and when in doubt use the defaults. If you’re using L1/L2 regularization, make sure your weight decay value is not set too high. If it is, your model might instead be optimizing for reducing the values of its parameters to 0 instead of achieving good predictions.



loss function

make sure you’re using the correct loss function for your use case (MSELoss I imagine).


monitor the gradient, see if it is not vanishing/exploding.