This answer perhaps comes too late for you, but the major use case of pooling is that when you shrink the data then a convolutional layer will learn higher level features (i.e. pooling aggregates information).  E.g., your (few) first high-dimensional layer(s) will learn small edges, curves etc; your pooled layer will learn lines and shapes; and finally further pooled layer recognizes whole objects. So it is not only for performance, but to actually learn