Hi tom, thanks for your reply. I would like to do more description about my solution.
Imaging the input (x) shape is N * C * L and it is padded with zero on invalid position. Meanwhile, there is a “0/1” mask (x_mask) with shape is N * L. In the mask, 0 means padding and 1 means valid position. After convolution, the output (y) shape will be N * C’ * L’ and the mask (y_mask) shape will be N * L’. To get y_mask, I have to compute the change of valid length for every sample in the batch. This step is very inefficient and less elegant in my point of view since there is a loop for N. Finally, I will do y * y_mask to get new input for the next step and I don’t need to worry about the bias. Pseudo code is seen below.
def forward(self, masked_x, x_mask):  # masked_x's shape is N*C*L and x_mask' shape is N*L
    y = self.conv(masked_x)  # y's shape is N*C'*L'
    if mask is not None and self.size_changed:
        x_valid_len = mask_2_size(x_mask)  # x_valid_len's shape is N*1
        for n in x_valid_len[0]:
            x_valid_len[n] = self.output_size(x_valid_len[n])
        y_mask = size_2_mask(x_valid_len)  # y_mask's shape is N*L'
        y = y * y_mask.unsqueeze(1).expand_as(y)
    else:
        y_mask = x_mask
    return y, y_mask

The mask_2_size, size_2_mask and self.output_size are functions I wrote.
Thanks for your example but I think it isn’t appropriate for my work, or I got your idea wrong.
I would be very grateful if you can give me some examples in pseudo code.