Perhaps you are running into this issue: https://forums.developer.nvidia.com/t/xid-8-in-various-cuda-deep-learning-applications-for-nvidia-gtx-1080-ti/66433 29?
It seems like one of the underlying causes of that error is the cuda kernel taking too long due to the GPU being used for other processes (like driving the display), which starves your training process and eventually aborts it. Is that forum post applicable here?