I think the pooling operation won’t change anything, since (as you said) the activation contains only one pixel in the spatial dimensions.
Your approach will most likely work.
However, alternatively to the convolutional approach, you could try to swap the last max pooling layer for an adaptive pooling layer, where you can define the output size.
I’ve created a small example 11 some time ago.