I cannot reproduce the memory leak using this simple code:
device = 'cuda'
bn = ComplexBatchNorm2D(3)
bn.to(device)
x_r = torch.randn(10, 3, 24, 24, device=device)
x_i = torch.randn(10, 3, 24, 24, device=device)

print(torch.cuda.max_memory_allocated() / 1024**3)

for epoch in range(100):
    bn.zero_grad()
    output = bn(x_r, x_i)
    print('Epoch {}, max mem {}'.format(
        epoch, torch.cuda.max_memory_allocated() / 1024**3))
    output[0].mean().backward()
    output[1].mean().backward()

Could you take a look and give me a hint how your usage differs from mine?