Right. However, I don’t think such mechanism would be a good default. There
are also other optimizers that work with sparse gradients (eg SGD iirc) and
we expect to have more in future so restricting to using SparseAdam isn’t a
good choice.
That said, I agree that this is a bit clunky at the moment. Perhaps some
way to combine optimizers into one will be good.