Solved by leowalkling in post #4 
              
              
                For a similar problem, have a look at https://github.com/leowalkling/sparsemap/blob/master/python/sparsemap/layers_pt/base.py (a fork from vene/sparsemap)