Well I know GPUs run slow when they are doing large small data sizes and few calculations will run faster on a CPU over a GPU.
From game dev a GPU is much better for larger number and a lot of calculations over and over again. That is why Deep Learning models like the gpu. The tuning of weights are many many small calculations that can happen all at once as GPU have 1000 of cores. A cpu can only do one thing at a time in each core.