Solved by albanD in post #4 
              
              
                IIRC, NHWC allows you much better implementation for convolution layers that gives a nice boost in perf (because the value for all channels is accessed for every pixel, so data locality is much better in that format). 
The problem is that batchnorm-like layer are so much faster in NCHW, that vision …