Interesting. I’ve tried changing that for a large network, so I can’t say what specific size it was slow for. Regarding slowdowns and autograd, what can you say about TensorFlow’s XLA performance? From benchmarks I’ve seen it seems to deliver quite fantastic optimizations. And that’s with partly due to optimizing things like relu (to cut on memory movements in this case), right? Relus and stuff seem to matter much more in the Tensor Core days than they did before Volta/Turing, especially on unlocked GPUs (not 20xx, 30xx series).