When I ran my code with Pycharm, it often gives me a error “CUDA out of memory…”. So I tried to find the reason by code debugging. Mostly, it requires 4513m memory to run the code, but when it comes to the first line (i.e., s=1.0/(1.0+torch.sum((c.unsqueeze(1)-self.msfcae.scale1_mu)**2,dim=2)/self.opt.alpha)) of the following function, the memory will rise to about 10613m. Then I added a line of ‘torch.cuda.empty_cache()’, memory usage will reduce to 7917m, but it will not fundamentally change the situation. It will also give an error in next epoch since the memory requirement is gradually increasing. I thought this may derived from the existance of broadcasting, since c.unsqueeze(1) and self.msfcae.scale1_mu have different shapes. By step-wise debugging, I found torch.sum((c.unsqueeze(1) - self.msfcae.scale1_mu) will increase the memory requirement. Then the square ** will further increase the memory requirement.  I do not know why the first line of this function will cost much memory, and could anyone please give me some suggestions to avoid this? Thanks a lot!
def scale1_soft_assign(self,c):
        s = 1.0 / (1.0 + torch.sum((c.unsqueeze(1) - self.msfcae.scale1_mu) ** 2, dim=2) / self.opt.alpha)
        torch.cuda.empty_cache()
        s = s ** (self.opt.alpha + 1.0) / 2.0
        s = s / torch.sum(s, dim=1, keepdim=True)
        return s