You could push the data and pooling layer to the GPU for a potential speedup.
Note however, that you might fallback to the native im2col implementation for this particular use case, if e.g. cudnn cannot find a fast kernel for this workload.