The CUDA API is asynchronous in nature. Maybe you are not actually getting the real runtime.
Run your program with:
CUDA_LAUNCH_BLOCKING=1 python my_program.py

This will for the CUDA API to be synchronous for the lifetime of the program.