If you are doing deep learning, PyTorch will be way easier than trying to do everything in raw numpy.  And unless you are using really tiny networks, you will want to use a GPU.  There might be some deep learning library out there that supports non-CUDA GPUs, but currently NVIDIA GPUs are so dominant in deep learning that you should just get one or you will be fighting an uphill battle.