As far as I know the mathematical/conceptual way of doing it is layer by layer. This is because different input image sizes will have different output shape i.e. the output shape will be different for an input of size (3, 128, 128) than for an input size of (3, 1024, 1024). There is no generalization because you will always have the variable of the input size. But if you find out a way I would also like to know it