if your project only has small computations, use CPU, otherwise use CUDA
Unless you are doing advanced code, stick to either using CPU for the entire project, or using CUDA for the entire project
depends on operation, have to benchmark and see
if model is very very small, they no point using CUDA

.cuda() and .cpu().
yes, sparse operations are usually faster on the CPU.