Well, if you want 0 to map to 0, I think you want signed integers rather than unsigned ones (which are the default), so use qint8 instead of quint8. Note that this may impact the operator coverage.
You probably know this, but just in case: for the sparsity to lead to less computation, you need special “structured sparse kernels”. I think they are being worked on, but it’s not what you get today.
Best regards
Thomas