So it is hard to comment on very concrete situations from somewhat more general descriptions, but here are some thoughts:
In general, it is a good idea to keep the difference between training and testing smallish.
In practice, the two most notable exceptions are

dropout, which has an interpretation as taking a mean during evaluation where during training we sample from a distribution,
and batch norm, where we don’t like the discrepancy between training and testing, so for example transformers use layer norm which, among other things, eliminates the train-test discrepancy even if it make evaluation quite more costly.

For other things, like spectral norm, we do don’t do all the compute as during training, but make sure the same weight is used in evaluation as in training.
From your description, it would sound like you are more in the spectral norm-like case. If you actually clipped/normalized weights, they might keep that so you would not need to do it over and over again, if you worked on the activations (intermediate results of the computation), you would keep it.
Best regards
Thomas