the short answer is that this is not really a simple effort, especially with autograd’s dependence on the base classes for type checks etc. We did do this with Sparse Tensors, but it wasn’t a trivial effort and we had to introduce C side base classes for sparse.
I’ll think about it and consult with the others and can get back to you.