It seems then that the number of convolutions/layers is a performance parameter that is essentially more important than the number of parameters and number of flops in a network - I definitely underestimated that.
So what would be my options if I wanted to make models with a large number of modules run more quickly, e.g. for a final implementation to be used only for evaluation on a robotic system?
A preliminary search turned up: https://pytorch.org/tutorials/advanced/cpp_extension.html
What other options would I have?