Unfortunately, that happens if your rendering uses the GPU and you do GPU compute at the same time.
What I used to do on my local machine is to get an old and cheap GPU that I use only for rendering the screen: attache your screen to it and hide it with CUDA_VISIBLE_DEVICES=1 when doing pytorch job.
That way the two will be independent.