During training the batch norm layers will update their running estimates of the mean and std and just apply these estimates during validation and testing.
The momentum term defines the update step of the running estimates using the current batch statistics.
We could see some issues (bad testing accuracy while training looked good), when these estimates are off due to e.g. different data domains for the datasets, and might need to resample/shuffle the data or change the momentum term.
That being said, I don’t want to say that your approach is invalid in any sense.
If you have valid assumptions, how you would like to apply the whitening during training and testing, I would recommend to run some tests and see, how it compares to e.g. standardization and batch norm.