Short answer is try both and keep the fastest 
Longer answer is that moving stuff from and to the gpu is quite expensive.
So if your other operations are really large and benefit a lot from the gpu, then it is worth doing them on the gpu even though you pay the transfert. If your other operations are quite cheap, then the transfer is going to be more expensive than doing the operation directly on the cpu, and so you should just do everything on the cpu.