Exploiting sparsity in batch operations?
If I have a batch of sparse input matrices, and a dense batch of matrices : <SCODE>> mat1 = torch.zeros(4, 3, 5)
> mat1[1][1][1] = 1; mat1[2][2][2] = 1
> mat2 = torch.rand(4, 5, 6)
> torch.bmm(mat1, mat2)
<ECODE> Exploiting sparsity is quite an optimisation. If this isn’t available yet, which users might I liaise with to help out?
cjmcmurtrie
Also, torch.smm(mat1_i, mat2) is also implemented, for sparse * dense -> sparse operations.
ebetica
<SCODE>import torch
import torch.autograd as ann


mat2 = ann.Variable(torch.rand(4, 5, 6), requires_grad=True)
mats = [ann.Variable(torch.zeros(4, 3, 5), requires_grad=True) for _ in range(3)]
for i in range(len(mats)):
    result = torch.bmm(mats[i], mat2)
    print result.size()
<ECODE>
cjmcmurtrie
Something like this: <SCODE>import torch

x = torch.rand(5,6)
# Sparse matrix of (0, 1) = 1; (2, 1) = 2, (3, 4) = 3
sparse = torch.sparse.FloatTensor(
    torch.LongTensor([[0, 2, 3], [1, 1, 4]]), # Indicies
    torch.FloatTensor([1, 2, 3])) # values

print(x)
print(sparse.to_dense())
print(torch.mm(sparse, x))
# This won't actually save space or compute, since it's so dense,
# but it will be a sparse tensor representation.
print(torch.smm(sparse, x))
<ECODE> Simply construct a list of your sparse tensors, and loop over them to do the batch mm.
ebetica
Thanks once more! One last thing: <SCODE>sparse = torch.sparse.FloatTensor(
    torch.LongTensor([[0, 2, 3], [1, 1, 4]]),   # Indicies
    torch.FloatTensor([1, 2, 3]))   # values
<ECODE> Seems strange to me as you don’t define the sizes of the sparse matrix - it seems to arbitrarily pick the indices of the corner value as the size. What is the logic here?
cjmcmurtrie
ebetica
cjmcmurtrie
By the way, is anyone working on wrapping cusparse?
Ilya_Kostrikov
martin raison is working on it, are you interested in collaborating?
smth
Yes, sure! I’m not sure whether I’m familiar with pytorch internals enough to be able help. But I can try anyway.
Ilya_Kostrikov
I’d like to collaborate on writing a wrapper for cusparse, if you folks still need a hand.
siddharthachandra
smth
Thanks, looks good. Let me check it out.
siddharthachandra