You should be able to use any Python object in your code, which works on the CPU, as long as the actual tensor is pushed to the device.
CUDA kernels are executed on the tensor data directly, so no Python dicts etc. are passed to the kernel.