@vmirly1 Thank you for the detailed response. This makes perfect sense. In fact this is what I was thinking as well. But I was not sure.
Now if there are some developers around, would it be too much of a complexity to abstract away the conversion ? So that one could use both the  convolutional and linear layers in a same Sequential module.