Is it perphaps this https://github.com/pytorch/pytorch/blob/cfe3defd88b43ba710dd1093e382c5e8c279bd83/aten/src/ATen/native/sparse/cuda/SparseCUDATensorMath.cu#L147 7 that we’re seeing take up time in forward in our original case? This would partially explain why we only see backward taking longer in the toy example (because to_sparse returns a coalesced tensor?). But it rises a question, if this is indeed the issue, then why is the tensor not coalesced during backward?