Basically, you can call any maths functions in cuda. FP32/INT32 is reasonably fast, fp64/int64 (for indexing) is significantly slower
I would recommend looking at CUDA programming patterns for typical ops, e.g.

pointwise operations (the easiest),
reductions (e.g. the combination of warp shuffles and shared memory),

The other part to know is how global / local memory and caching works.
Also look at code, PyTorch has implementations for many ops, so does torchvision for the detection ops.
There are also some resources looking at kernels for PyTorch, e.g.  I discuss the a reduction pattern on my blog 4. This isn’t terribly efficient for matrix multiplication-like patterns, but it’s faster than CPU. 
Best regards
Thomas