@ptrblck could you please explain where the matrix m1 [1x20592 ] is coming from. Because  output after convolutions should be 322 311 = [1, 20526]  after flattening.
I even tried it out on dummy data with the same shape and the output tensor is as expected