Yes, the explosion in memory is torch.cat() growing the number of connections.
What’s happening in this specific case is that you’re calling torch.cat with a Variable argument instead of an iterable of Variables. torch.cat indexes its input a few times which adds to the computation graph.
Here are my profiling results (I’m not sure why my numbers are 10x less than yours, but the ratio between the memory consumptions seems okay):

(129mb) First code block, on my machine
(1.04gb) Code block with cat as you wrote it. This looks bad, as you mentioned.
(411mb) Replaced torch.cat((x[:, i, :]), -1).view(60, 100) with x[:, i, :].clone(). This is what the ideal memory use should be close to because all the input's are staying around in memory.
(415mb) Replaced torch.cat((x[:, i, :]), -1).view(60, 100) with torch.cat([x[:, i, :]]. This is what I would advise changing the code to to avoid the memory jump.

The advice here is that the argument to torch.cat shouldn’t be a Variable, it should be an iterable of Variables.
However, you probably aren’t actually calling torch.cat with a Variable because that doesn’t do anything. Could you provide a more detailed example of what you’re doing and how the memory is blowing up?