Yeah, that sounds also reasonable.
However, some of my models were just killing the complete input signal (making it negative, so that relu will set it to zero), while the last output bias was creating my prediction.