I think there are several parts:

I haven’t implemented SAGAN yet, but SNGAN doesn’t use eval mode at all when training the generator. I think that is mostly because of batch norm which behaves quite differently between training/eval.
I would expect that for a fully trained network, the impact of additional updates of the spectral_norm don’t do that much. You could see what the largest eigenvalue of the weight with spectral normalization . Also spectral normalization affects only the discriminator (at least in SNGAN)?
I’m surprised that in training mode the behaviour of batch norm changes: While the statistics are updated, they are not used in training. The weight/bias parameters on the other hand should not be updated without calling optimizer steps.
When you are just sampling, you don’t call the optimizer’s step, do you?

You could experimentally try to get closer to the matter:

If you just set the batch norms to eval, does it output garbage, too?
Does generating samples involve spectral normalization?
If you set everything to eval except the batch norms, does it work? (Then you don’t need to worry about spectral norm.)

Best regards
Thomas