Hi,
That would depend a lot on your forward pass. It should not do that 
Do you have a code sample that repro this? If possible on colab (https://colab.research.google.com/notebook#create=true 8) so that we can easily test it.