Hi,
My model has a weight tensor W, which contains duplicate entries due to symmetry.
I compacted the weight tensor as U which is 1/4 the size of W, and generate an index tensor B (same size as W) to deflate it back to W on the fly, as W = U[B], note that B’s content is static/fixed.
I was hoping that since B is generated on the fly, we can save some memory, but training in Pytorch actually uses more memory.
What is the reason behind this? If index B is cache because it can be predetermined, can I force it to be always generated on the fly?
Thank you very much!