Thank you for the quick reply.
I use a Jetson AGX Xavier with CUDA 10.0 in combination with ROS which unfortunately limits me to this configuration including Python 2.7. I will maybe build a newer version with Pytorch from source and try to setup ROS with Python 3 if I have time. However, I will first investigate if this is due to the Pytorch version or maybe Python 2.7 itself. I will investigate this and report back.
For now I fixed this leak(?) by splitting up the matrix multiplication in a for loop and using a max size for n of 30 in each step. This does not result in RAM memory building up until overflow.