One thing to point out is to add a if statement like below to make it robust to padding=0:
      if self.causal_conv.padding[0] != 0:
         x = x[:, :, :-self.causal_conv.padding[0]]  # remove trailing padding