@Tony_Lee in short yes but it’s not that simple.
Although GPU specialises in parallel computation it can only perform vectorized math and in this case tensors. And not general purpose that need sequential work, even callback mechanics might misbehave with GPUs. Also each individual cores of GPUs are significantly weaker than CPUs.
Although in case of GPU programing we tend to prefer high level APIs and avoid programing dealing with thread, processes and so on.
Here’s an article nicely put by NVIDIA.

  
      The Official NVIDIA Blog – 16 Dec 09
  
  
    

What's the Difference Between a CPU and a GPU? | The Official NVIDIA Blog 13

What is a GPU? And how is that different from a CPU?