There is some silly calculation mistake I am making in calculating the in_features of fc1 which I am not able to catch. The above code only works for odd kernel sizes so for now I just went ahead it with.