Yeah, but the op before is the backward for nn.BatchNorm1d, which should support sparse gradients. Ok, I guess I need to dig a little more then. Thanks so much. Will write here, if something interesting is found.