Hello Eta_C, this is much faster indeed, thank you!
Turns out that our actual bottleneck was not here though, it seems to be in the backward pass, which we didn’t really see in our profilers, I guess because of asynchronous operations, so it’s all good.