It should generally be working. However for most network architectures you’ll get into trouble regarding the shapes of the outputs of your intermediate layers.
In the Inception paper 3 they added an auxiliary path so that the output shapes were reasonable to compute the loss.
If you are just dealing with a fully convolutional model without changing the spatial size too much (as shown in your example), it should be alright.