@ptrblck the full precision network is producing state-of-the-art accuracy. However, the XNOR version is the issue. The gradient flowing through is always zero. Now, if I remove the permute operation and just treat is as 1d, it works, the gradient is not zero, however, it is not the same network, right.