Ok, so it is inevitable to allocate new memory when concatenate two tensors in pytorch right now.
Is it possible to implement a new concatenation operation like this post 203 in pytorch? It records the reference instead of memory copying.
This new feature should be pretty useful I think.