Solved by albanD in post #2 
              
              
                The computational graph is not linked to the model, a new one est created each time you perform a forward pass. So no it is not overriden and yes the memory usage increase as you create 3 computational graphs (each containing its own intermediary results needed for the backward pass).