I found out the problem! During my training routine it would call validation sporadically but I forgot to set mode back to train once validation was finished. So basically I was training in eval mode without realizing it…
Thanks for the help!