I can’t imagine how two separate biases for z and r would increase efficiency or performance in any way. The gradients will be identical. Perhaps the new gate might* have some nominal additional information from two separate biases. as one is added prior to the Hadamard product. But I wonder if it was tested first for any performance gains. Or if, as you mentioned, the underlying architecture in cudnn just made it a better choice to leave in. That’s good to know. But then I wonder how Tensorflow got around that with their v3 implementation.