As far as I understand your use case, you are creating a batch (3-dimensional: [batch_size, channels, seq_len]), where some tensors were zero-padded in the last dimension.
Is that correct?
Now you would like to ignore the padded inputs in the batchnorm layer, i.e. not being taken into account for the running stats or what would the desired behavior be?