The gradients might still be needed if previous layers depend on them.
To disable the stats updates, call .eval() on the batchnorm layers or use track_running_stats=False if you want to use the batch stats during training and evaluation.