I forgot exploding gradients was a thing… After clipping my “o” value the model now has gradients but I’m not sure if they are vanishing or if my model is wrong because it no longer changes after one step of SGD.