Seems like the problem is closely related to my input tensor size. Each time there is a input tensor which is larger than any tensor before the cached memory size will increase significantly. The memory leaking issue happens at three place. First is at out = self.param(embed) in forward function. Second increase happens at loss = criterion(ret, labels). The third increase happens at loss.backward().
So it appears that once I get a tensor input which has the largest size so far during training, the cached memory size will increase for every operation after a linear transformation. Sometimes the cached memory get released when GPU is running out of VRAM, sometimes Pytorch just throws an OOM error and get aborted. What might be the reason for this?