I wrote to the authors regarding this very problem.
Turns out they just push the entire image through the network. I just tested it again and it works all fine. Previously when I tested it I might have forgotten to properly conver scales (from 0-255 to 0.0-1.0 for instance) or something. So during evaluation the input image can indeed be much larger than 40x40 pixels. Well, actually not that large because when I try passing through a full HD image my GPU runs out of memory (8 GBs). So I think that in the end I will still have to segment the input.