This is not the problem. Backprop only works using the loss and the existing weight tensors of conv layers in this case. It does not work with the input dimension, data dimension of batchnorm, activationa and others. This network works as expected as a full precision network already.