You hinted at some more advanced techniques @jmaronas . I feel like if there was any way to speed things up it would be included in pytorch already. Because all deep learning is doing the same matrix multiplications. So I feel like there’s not many practical gains I can make by going deeper below the pytorch abstraction layer
I was looking into nn.DataParallel https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html 34