I currently have about 1 million data points with 3000 sparse features. At first, I thought that PyTorch sparse tensor would be useful in this case, but I noticed that data loading on sparse tensor was very slow while using dataloader.
Here’s an example on my current situation.
import time
import torch
from torch.utils.data import TensorDataset, DataLoader

x = torch.FloatTensor(800000, 300).random_(0,5).to_sparse()
y = torch.FloatTensor(800000, 1).random_(0,5)

train_ds = TensorDataset(x, y)
train_dl = DataLoader(train_ds, batch_size = 1024)

start_time = time.time()
for x,y in train_dl:
    print("--- %s seconds ---" % (time.time() - start_time))
    break

# Output:
# --- 497.44726943969727 seconds ---

As you can see, the data loading on sparse matrix was very slow (1 batch used about 497 seconds) and it is even worse in my use case with 3000 sparse features.
So is there a way to speed up data loading on large sparse matrix while using dataloader? Thanks