You are using option 2 from there right?
Do you see the memory usage increase at each iteration?
Also depending on which optimizer you use it might need to allocate a fairly big state. You need to make sure that 1step + optim works fine before doing more steps.