Did you profile your code to see where the bottleneck is?
It might be that e.g. your data loading is the bottleneck so that reducing the operations in your model won’t change anything.
Also, did you time the linear vs. pooling layer properly and made sure the pooling layer is indeed faster?