Is it possible to lazy-generate a larger-than-memory random matrix with pytorch? Ideally I am looking to generate a 1e9 x 5000 matrix and compute the scalar product with another 5k x 5k matrix.
I had a look at keops, but I didn’t manage to get too far with that.
Is it be possible to do this in a distributed manner, for example by interfacing somehow with pyspark?