Well it depends on the field and the task. You have to think in the real pipeline.
For example, generating speech from text.
Do you have access to the whole sentence? Yes, thus you don’t need to mask that.
Do you have access to the whole speech? No, because you are predicting it. Thus you have to mask it.
The key is you (usually) don’t need masks at inference, as you predict one element and feed that and all the previous.
But the best you can do is looking for code  that always help. No matter it is not using pytorch’s transformer, the key idea is the same.