I am doing a correspondance calculation between two matrices. One of these vectors has a fixed shape of [256, 5000] and the other one is variable of shape [256, n]. Here n is usually smaller than 100.
Both matrices are on the gpu and I do the calculation as follows:
 # mat1, mat2 are on cuda
correspondance_mat = mat1.unsqueeze(-1) * mat2.unsqueeze(1) 

This results in a matrix of shape [256, 5000, n].
The strange problem I am having now is that every time n increases to a larger value than ever before, a huge increase in CPU RAM occures. This used memory also does not seem to be realeased anymore, and it looks like its leaking. This even gets my computer to freeze when it reaches 100% memory occupation.
My question now is what is happening here?
1.) Since both matrices are on GPU why does it even affect my CPU RAM???
2.) Why does it not release the memory anymore after it is finished with this multiplication?
3.) Could this be a bug/memory leak in pytorch?
My Pytorch version is 1.4 and I am using it with Python 2.7. Unfortunately there is no way I can upgrade this version.