I am plotting the gradients. It seems that removing conv layers after permute and keeping only the linear layer does not make the gradient to be always zeroes. It becomes all zeroes when there is even a single conv layer after the permute. Does that mean, somehow the back-propagation is failing to consider the permute operation? If that is the case, then how the full precision network is working? No clue so far.