After going through some testing I had to find out that while the solution now looks cleaner with the cat() instead of the resize_() as there is less lines, it still actually has the same memory problem.
Given that cat() is not an in-place operation if I do something like:
a = torch.cat((a,b), dim=0) 

it seems to internally create the memory copy now. To give an extreme example to illustrate the point: If I have a 11Gb GPU,  a is 10 Gb in size, but b is only 10Mb, it runs out of memory in the cat operation.
This seems equivalent to the resize_() method that operates in-place and filling the resized tensor with a clone(). I couldn’t find a source implementation for cat(), but it looks like this is what it might do internally after all.
So unfortunately, my first question remains open. I will have to think further about your latest post, but it seems really difficult to implement if one repeats addition of units to the layer many times.