it has high chances of zeros present in input data.

Ideally the result shouldn’t be dependent on the length of the padding.
In some cases tf.ragged can make this sort of code easier to write.
It may not be the masking causing the poor performance.