For any future developers that come upon this, I figured out why, I had some dropout based layers in my code, which means the network being in train/eval mode mattered.
I was able to fix this by calling
model.train() in the beginning of my training function and model.eval() when calling just trying to get an accurate evaluation of a boardstate.