I was training something that looks like a transformer, and of course, this requires that you accumulate a tensor of previous tokens/other things in order to compute attention. This is facilitated by torch.cat() at each timestep,  which i have found slows down my code by a factor of 2, while also increasing memory usage by a factor of 2.
It makes sense that making new arrays is differentiable whereas inplace assignment to a preallocated array is not, but is this really the best we can do?! Since so many GPU hours are spent on training transformers, i find it very hard to believe that such a huge bottleneck like this is so widely adopted, and nobody is looking into alternatives.