Yes I think you might be right. I’ve never used track_running_stats before and I see it’s described as:
track_running_stats: a boolean value that when set to `True`, this
            module tracks the running mean and variance, and when set to `False`,
            this module does not track such statistics and always uses batch
            statistics in both training and eval modes. Default: `True`

It sounds like while it doesn’t track a running mean/variance, it is still using the batch mean and variance? In my case I think I would want a fixed mean of 0 and a variance of 1, but still allow for gamma and beta to be learnable parameters, correct?