Max only backprops through those layers which are maximum.
Anyway you have AdaptativeMaxPooling, which allows you to choose the final size. That’s flexible enough to perform any operation I guess.
Does it fit your scheme?