So is it safe to say autograd is the computationally expensive aspect of training? What if I wanted to ship a custom forward/reverse function to a different host or cluster for offloading, instead of CUDA; is that a reasonable architecture for experimenting with non-GPU acceleration?
I have been looking at GPU performance with PyTorch on my GTX1060 here, and it doesn’t seem like PyTorch is achieving high GPU occupancy. Or at least my numbers are showing less than ~35% GPU utilization, and I am able to train two or three models at once before I get close to 95%+. So I am wondering if I would be able to get close to GPU acceleration with MPI for example, if I had a cluster to offload a custom forward/backwards function to over the network.