Hi, everyone. I find that GRU model does not free the cached memory automatically. Here is the code sample I use.
import torch
data = torch.rand(998, 400, 23).cuda() 
# 1183M memory usage
model = torch.nn.GRU(23, 128).cuda()
# 1247M memory usage
with torch.no_grad():
    h, o = model(data)
# 2811M memory usage
torch.cuda.empty_cache()
# 1443M memory usage

import torch
data = torch.rand(998, 400, 23).cuda() 
# 1183M memory usage
model = torch.nn.GRU(23, 128).cuda()
# 1247M memory usage
h, o = model(data)
# 2811M memory usage
torch.cuda.empty_cache()
# 2613M memory usage

Since I call the mode within the no_grad context, I expect no intermedium nodes will be created. But the result memory usage is no effect by the context. I wonder it is a feature or a bug for PyTorch? If it is a feature, how these cached memory will be used in the future?