Solved by mohaimen in post #21 
              
              
                So, I found the problem and solved it. Before Permute the tensor is [64, 32, 1, 151] and after Permute the shape becomes [64, 1, 32, 151]. Hence, the channels become 1. Here, the mean centre fails. I am avoiding mean centre for this particular layer. This solves the issue.