The “unhandled cuda error” is more of a generic error message - the actual error may be indicated by the “Cuda failure ‘invalid device function’”.
These errors are usually caused by some incorrect configurations, and you may try setting the following:
* NCCL_TREE_THRESHOLD=0
* NCCL_SOCKET_IFNAME=
* NCCL_IB_DISABLE=1
* NCCL_P2P_DISABLE=1
A similar error from Horovod (link here: Cuda failure 'invalid device function' · Issue #1171 · horovod/horovod · GitHub 3) was fixed by setting NCCL_IB_DISABLE=1