This is because the algorithm of the CPU and GPU implementation are different. I think the CPU implementation is based on the cross-correlation algorithm you often find in textbooks (i.e, the discrete version of it). The GPU approximates convolutional operations using fast fourier transforms.  think the GPU/CUDA is using some version of winnograd’s method.
E.g., a good overview can be found in Lavin, A., & Gray, S. (2016). Fast algorithms for convolutional neural networks. In  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition  (pp. 4013-402), https://arxiv.org/abs/1509.09308 2