As far as I understand, you should at least retain the graph of B since pytorch saves intermediate tensors in that graph, not in components of nn module.
If your loss is really constructed with form of A + B, you can call A.backward() and B.backward(retain_graph=True) separately.