Solved by simaiden in post #2 
              
              
                If you use a Linear layer you have to use fixed input size. If you want to use variable input size you may use 1x1 convolutions