When you say the value doesn’t change at all, you mean after doing the optimizer step?
If so yes, that might be due to Adam that regularizes a lot the trajectory if it is very noisy. You can try with SGD to see if you get the same behavior.