General question. I ran an experiment to test the number of bytes used in sparse matrices versus nn.Linear layers of the same size. I set both nn.Linear layer and sparse matrix to all zeros and found that nn.Linear is twice as large as a sparse matrix. Why is that?
1 x 1,000,000,000 matrix
nn.Linear			8.0 Gb
sparse.FloatTensor	4.0 Gb
You can find my code here 4