I found the problem myself.
I was not changing the correct variables in the GRU layer. I had not understood the variables-names properly, as they are described in GRU — PyTorch 1.7.1 documentation 1.
Changing to
gru.bias_ih_l0[:]=0
gru.bias_ih_l0_reverse[:]=0
gru.bias_hh_l0[:]=0
gru.bias_hh_l0_reverse[:]=0

solves it.
(I discovered it by inspecting a gru object directly)