It seems that the memory usage is pretty constant for some iterations. The initial peak usage might be caused by creating some temporal objects, which might have been deleted later.
Are you using some kind of variable shape in your model, which could increase the memory after 507 steps or is this particular iteration somehow “special”?