in optimized implementations it is not sliced, instead batch matrix multiplication is done: (1000,8,96) @ (96,hs*3) = (1000, 8, hs*3) = (1000, 8, hs*3), this is Wx summands for three gates and all timesteps at once (hs=hidden size, it can differ from input size 96) . This is a “precalculation”, as Wh summands must be computed step by step.