Thanks for the code!
I cannot reproduce a memory leak with a master build + CUDA10.2 or with the 1.4.0 binaries + CUDA10.1.
I’m using your model and this code:
model = MyModel().cuda()
x = torch.randn(1, 16, 115, 115, 115).cuda()
torch.cuda.synchronize()
print('mem allocated ', torch.cuda.memory_allocated()/1024**3)

for idx in range(100):
    output  = model(x)
    print('iter {} mem allocated {}'.format(
        idx, torch.cuda.memory_allocated()/1024**3))

Do you see a memory leak using my test code?
PS: I tested the volumetric shapes of 110, 115, and 125.