Thank you for the advice. I thought what permute can do is switching the dimension of tensor. I try to print the size of input and I got
starting training
torch.Size([1, 1, 16, 184])
loss : 6.1917, time : 0.4552
torch.Size([1, 1, 16, 148])
Traceback (most recent call last):
  File "maintxt1.py", line 241, in <module>
    output = model(input)
  File "/home/tangyeping/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tangyeping/pixel-cnn-pp/model.py", line 119, in forward
    x = x if sample else torch.cat((x, self.init_padding), 1)
RuntimeError: Sizes of tensors must match except in dimension 3. Got 184 and 148

I am not sure how I can use permute to reshape the data like what transform.Resize() does.