As you might expect, @ptrblck 's explanation totally nails it, I’d probably strike the “almost” in almost zero influence.  The gradients should be 0 here as the result does not depend on the parameter. In other words, use bias=False for the linear/conv preceding batch norm.
If it is any consolation, it is #5 on Andrej Karpathy’s somewhat famous list of common NN training mistakes 4, so you are in good company.
Best regards
Thomas