Hi Kushaj,
Thanks for the reply! I did used a “shuffle=True” in my dataloader. I also checked the max absolute values for data points and the data is normalized to N(0, 1).
On another note, your idea makes me think could this because of I have a small dataset? There are only 947 samples and my minibatch size is 25. Maybe one dominant datapoint just somehow changes the loss too much.
I’ll continue playing with my network and update here if I found anything. Thanks!