And if so, why. We should at least be able to avoid matrix multiplications and just sum the biases. There should be a shortcut.
Note: I am not talking about ‘mostly zero’ sparse matrices. I am talking about something more fundamental; you do not need to multiply by a matrix if the input is zero.