In typical case, for each batch, a new graph is created, after the backward pass, the graph is freed. Then next batch comes, and new graph is created. The process repeats until you finished your training.
For more info, you can read my post 564 on computaion graph in PyTorch, hope it can help you.